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EXPERIMENTAL  STUDIES  OF  CAD A- BAS ED 
UTILITY  ASSESSMENT  PROCEDURES1 
Melvin  R.  Novick 
Nancy  J.  Turner 
Laura  R.  Novick 
The  University  of  Iowa 


Several  experiments  were  performed  to  investigate  the 
degree  of  incoherence  associated  with,  and  the  extent  of 
biases  caused  by,  the  certainty  and  anchoring  effects  for 
the  following  utility  assessment  procedures:  standard 
gamble  fixed  state  (without  and  with  overfitting  and  least 
squares  Incoherence  resolution),  regional  coherence,  and 
local  coherence.  Several  elicitation  formats  were  also 
compared.  The  purpose  of  these  experiments  was  to  assist 
in  the  development  of  a  reliable  (coherent),  bias-free, 
CADA-based  method  of  utility  assessment  for  general  use 
in  educational  evaluation. 


Utility  assessment  can  be  treated  as  a  topic  in  the  classical 
psychometric  specialty  of  scaling  (e.g.,  Torgersen,  1958)  or  more 
formally  as  part  of  a  theory  of  measurement  (Krantz,  Luce,  Suppes, 
and  Tvcrsky,  1971).  Novick  and  Lindley  (1978,  1979)  have  dealt 
with  the  use  of  utility  functions  for  applications  in  education  and 
have  advocated  the  use  of  the  standard  gamble  (von  Neumann  and 
Morgenstern,  1953)  elicitation  procedure  with  the  addition  of 
coherence  checking  using  overspecification  and  a  least  squares  fit. 

In  this  procedure  utilities  are  Inferred  from  probability  judgments 
offered  by  assessors. 

^This  research  was  supported  under  Contract  #N00014-77-C-0428 
between  the  Office  of  Naval  Research  and  The  University  of  Iowa. 
Opinions  expressed  herein  are  those  of  the  authors  and  do  not  neces¬ 
sarily  reflect  those  of  the  contracting  agencies.  The  computer  program 
development  of  David  Libby  and  the  consultative  assistance  of  David 
Chuang  are  gratefully  acknowledged. 


2 


j 

Earlier  approaches  to  utility  assessment  (Hosteller  and  Nogee ^ 

1951;  Schlalfer,  1959,  1971;  Raiffa  and  Schlalfer,  1961;  Heaney  and 
Raiffa,  1976;  and  so  on)  have  been  based  on  the  use  of  fixed  proba¬ 
bility  assessment  procedures  in  which  utilities  are  elicited  directly. 

It  has  been  suggested  (Hosteller  and  Nogee,  1951)  that  such  pro¬ 
cedures  are  easier  to  use  because  subjects  are  generally  more  familiar 
with  the  quantity  for  which  the  utility  function  is  desired  than  tkey 
are  with  probabilities.  Since  use  of  an  easier  procedure  does  not  necessarily 
Insure  that  subject's  responses  will  be  coherent,  an  opportunity  for 
incoherence  resolution  is  needed.  However,  the  nature  of  the  fixed 
probability  gambles  makes  it  very  difficult  to  check  for  coherence  of 
responses. 

Although  it  was  originally  thought  that  utility  theory  would  prove 
useful  as  a  descriptive  model  (Swalm,  1966,  etc.),  much  criticism  has 
recently  been  levied  against  the  use  of  utility  theory  in  that  capacity 
(Coombs,  1975;  Kahnesum  and  Tversky,  1979).  As  principal  critics, 

Kahneswn  and  Tversky  have  proposed  an  alternative  descriptive  model. 

The  mein  basis  for  their  criticism  is  that  the  phenomenon  described  by 
Tversky  (1977)  as  the  certainty  effect  results  in  preferences  that 
violate  the  substitution  axiom  or  expected  utility  hypothesis  of 
utility  theory.  This  axiom  (hypothesis)  ststes  that  preference  order  is 
invariant  over  probability  mixtures  and  is  formally  equivalent  to  the 
assumption  that  there  is  no  positive  or  negative  utility  for  the  act  of 
gambling  itself.  Specifically,  the  certainty  effect  is  the  phenomenon 
that  the  utility  of  an  outcome  seems  greater  when  it  is  certain  than 
when  it  is  uncertain.  This  effect  can  be  observed  when  subjects  are 
presented  with  a  choice  between  a  for-sure  and  a  chance  option,  the  choice 
presented  in  the  standard  gamble,  regional  coherence,  and  local 
coherence  assessment  procedures  discussed  below. 
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Although  utility  theory  is  considered  here  es  s  nonset ive  model 
-Tether  then  es  a  descriptive  model,  it  is  still  important  to  con¬ 
sider  the  certainty  effect.  Tversky  (1977)  has  shown  that 
even  when  subjects  were  told  that  their  preferences  violated  uti¬ 
lity  theory,  they  were  not  inclined  to  change  them  (see  also 
Kahneman  and  Tversky,  1972).  This  brings  into  question  the  relia¬ 
bility  (coherence)  and  bias-free  character  of  utility  assessment 
procedures  and  the  value  of  those  procedures  in  helping  decision  makers 
be  more  coherent.  However,  two  additional  comments  must  be  made. 

Whereas  the  gambles  studied  by  Kahneman  and  Tversky  involved  gains 
and  losses  from  some  undefined  reference  point,  those  studied  by 
Novick  and  Llndley  considered  passage  to  and  from  well-defined  states. 
Furthermore,  the  latter  authors  also  Included  incoherence  resolu¬ 
tion  as  a  part  of  their  utility  assessment  procedure. 

In  another  paper,  Tversky  and  Kahneman  (1974)  described  several 
heuristics  used  by  persons  in  assessing  probabilities  and  the  biases 
to  which  they  could  lead.  Of  particular  Interest  is  the  anchoring 
and  adjuatment  heuristic,  whereby 

the  most  readily  available  piece  of  information  often  forms  an  initial 
basis  for  formulating  responses,  from  which  subsequent  responses  are 
then  adjusted.  Since  adjustments  from  this  basis  are  often  insufficient, 
a  central  bias  results.  According  to  Slovlc  (1972),  the  anchoring  and 
adjustment  heuristic  is  e  natural  strategy  for  easing  the  strain  of 
integrating  information.  The  anchor  serves  as  a  register  in  which  one 
stores  first  impressions  or  the  results  of  earlier  calculations.  Slovlc 
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advances  two  hypotheses  to  explain  why  adjustments  from  the  anchor 
are  usually  insufficient.  First,  people  may  stop  adjusting  too  soon 
because  they  tire  of  the  mental  effort  Involved  in  adjusting. 
Alternatively,  the  anchor  may  take  on  a  special  salience, 
causing  people  to  feel  that  there  is  less  risk  in  making  estimates 
close  to  it  titan  in  making  sstimatea  far  from  it. 

In  reviewing  the  role  of  man-machine  systems  in  decision  ana¬ 
lysis,  Slovic,  Fischhoff,  and  Lichtenstein  (1977)  suggested  that 
human  factors  such  as  the  ways  in  which  variations  in  instructions 
or  informational  displays  affect  people's  performance  should  be 
studied  in  more  detail.  Some  progress  has  been  made  in  this  area 
with  the  finding  that  questions  of  complexity  and  representativeness 
of  material  seem  to  have  substantial  effect  on  assessors'  responses 
(Fischhoff,  Siovic,  and  Lichtenstein,  1977;  Vlek,  1973).  The  study 
of  such  factors  might  lead  to  an  assessment  procedure  that  minimizes 
the  judgmental  biases  and  heuristics  described  earlier.  This  posi¬ 
tion  was  8treughthencd  by  the  discussion  of  Fischhoff,  Slovic,  and 
Lichtenstein  <1979) . 

The  purpose  of  the  experiments  reported  here  was  to  assess  the 
reliability  (coherence)  and  degree  of  bias  of  several  proposed 
utility  assessment  procedures.  In  particular,  the  influence  of  the 
certainty  and  anchoring  effects  identified  by  Tversky  and  Kahneman 


(1977)  was  of  interest.  New  response  formats  end  recently 
proposed  locel  end  regional  coherence  (Novlck,  Chuaag,  end  DeKeyrel, 

1981)  procedures  were  considered.  The  ultistete  goel  of  the  experiments 
wes  to  assist  in  the  development  of  a  reliable,  bias-free,  CADA-based 
utility  eaeeesaent  procedure  for  general  use  in  educational  evaluation. 

Extensive  previous  work  in  this  area  has  raised  more  questions 
concerning  bias  and  coherence  than  it  has  provided  answers.  An 
apparently  pessimistic  mood  prevails,  not  inappropriately,  given 
the  importance  of  the  questions  that  have  been  raised.  Nevertheless, 
the  very  extensiveness  of  this  research  must  Itself  imply  a  high 
assessment  for  the  product  of  the  probability  for  resolving  these 
difficulties  and  the  value  of  this  outcome.  The  position  taken  here 
is  that  bias  and  incoherence  may  be  reduced  if  elicitations  are  care¬ 
fully  fashioned  in  a  Computer-Assisted  Data  Analysis  (CADA)  environment 
(Novick,  Hamer,  Chen,  Woodworth,  Libby,  Isaacs,  Lewis,  Holenaar  and 
Chuang,  1980)  and  assessors  are  aided  in  resolving  Incoherence. 

The  outcomes  for  which  utilities  were  sought  were  college 
grade  point  averages  (CPAs)  at  graduation  and  average  graduate  record 
examination  (CRE)  scores  of  first-year  graduate  students.  The  following 
fixed  state  utility  assessment  procedures  were  compared: 

\ 

1.  Standard  fixed  state  (SFS) 

2.  Regional  coherence  (BC) 

3.  Local  coherence  (LC) 
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The  order  of  the  seeps  taken  by  procedure  1  to  elicit  utilities 
(see  description  below)  made  possible  sn  evaluation  with  end  without 
incoherence  resolution.  The  following  formats  were  studied  for  ob¬ 
taining  indifference  probabilities: 

1.  Direct  probability  elicitation 

2.  Ends-in  method 

3.  Quartering  method 

4.  Modified  ends-in  method  with  graphic  representation 

In  addition,  three  sets  of  gambles  were  studied  with  the  SFS  procedure: 

1.  Adjacent  gambles 

2.  Distant  symmetric  gambles 

3.  Utility  gambles 

Experimentation  was  sequential  with  conclusions  drawn  and  insights 
gained  from  pilot  experiments  used  to  select  methods  and  formats  to  be 
tested  in  the  final  experiment. 

The  three  types  of  gambles  refer  to  how  the  CPA's  (alternatively,  GRE 
scores)  were  chosen  to  construct  those  gambles.  Adjacent  gambles  consisted  of 
three  successive  CPAs  at  O.S  intervals.  That  is,  the  three  CPAs  used  in  each 
gamble  covered  an  Interval  equivalent  to  one  point  on  the  CPA  scale — for 
example,  (2.0,  2.3,  3.0).  Distant  symmetric  gambles  used  successive  CPAs  at 
1.0,  1.3,  or  2.0  Intervals.  They  were  symmetric  in  the  sense  that  the  distances 
between  the  lowest  and  middle  CPAs  and  between  the  middle  and  highest 
CPAs  were  equal — for  example,  (2.0,  3.0,  4.0)  and  (0.0,  1.3,  3.6). 

Utility  gambles  derive  their  name  from  the  fact  that  the  indifference 
probabilities  ellcitud  for  each  gamble  are  identical  to  the  utilities 
for  the  for-sure  CPAs  of  those  gambles.  This  is  achieved  by  setting 
the  lower  and  upper  CPAs  in  the  chance  option  of  each  gamble  at  0.0 
and  4.0,  respectively.  The  following  are  examples  of  utility  gambles: 

(0.0,  1.0,  4.0)  and  (0.0,  3.3,  4.0).  Experiment  1  was  conducted  in  part  to 
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determine  which  gambles  In  each  set  produced  particularly  large  errors 
and  thus  should  not  be  used  subsequently. 

In  the  SPS  procedure  subjects  were  given  situations  consisting 
of  a  for-sure  and  a  chance  option  and  were  asked  for  the  chance  option 
probabilities  that  would  sake  than  indifferent  with  respect  to 
the  two  options  in  each  situation  (l.e.,  their  indifference  probabi¬ 
lities).  The  indifference  probabilities  for  the  fixed  state  gambles 
(situations)  could  be  elicited  using  any  of  four  methods,  corresponding 
to  four  formats  for  presenting  the  gambles,  and  the  determination  of 
the  format  which  elicited  the  most  coherent  set  of  utilities  was  of 
primary  concern.  Table  1  below  describes  each  format.  The  gamble  used 
in  the  table  can  be  abbreviated  as  (2.0,  2.5,  3.0)  and  this  notation 
will  henceforth  be  used  to  describe  fixed  state  gambles. 
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Format  One 


FOE  SUEZ  OPTION  CHANCE  OPTION 


I 

GPA  I 

I 

GPA 

I 

I 

I 

I 

3.5  P% 

I 

I 

2.5  I 

I 

I 

I 

I 

1 

1.5  100I-P 

I 

What 

value  of  p  makes  you 

indifferent  between  the 

for -sure 

and 

the  chance  options? _ 

_ 

o 

! 

ft 

Two 

OPTION 

I 

GPA  I 

I 

GPA 

I 

0 

INDIFFERENT 

1 

I 

I 

3.5  P 

I 

1 

FOR  SURE 

I 

FOE  SURE  2.5  I 

I  CHANCE 

I 

2 

CHANCE 

I 

I 

I 

1.5  1-P 

I 

3 

RESTART 

Which  option  would  you  prefer 

if  p  - 

.XX? _ 

Format 

Three 

FOR  SURE  OPTION  A 

CHANCE 

OPTION 

I 

GPA  I 

I 

GFA 

CHANCE 

I 

OPTION 

I 

I 

I 

3.5 

75% 

I 

1 

FOR  SURE 

I 

2.5  FOR  SURE  I 

I 

I 

2 

CHANCE 

I 

I 

I 

1.5 

25% 

I 

3 

RESTART 

Which  option  do  you  prefer? _ 

FOR  SURE  OPTION  ® 

CHANCE 

OPTION 

I 

GPA  I 

I 

GPA 

CHANCE 

I 

I 

I 

I 

3.5 

P 

I 

I 

2.5  FOR  SURE  I 

I 

I 

I 

I 

I 

1.5 

Q 

I 

Choose  che  set  of  P  and  Q  values  that  will  make  you  Indifferent 
between  the  for-sure  and  the  chance  options. 

P  75%  70%  65%  60S  55%  50% 

Q  25%  30%  35%  40%  45%  50% 


The  P  value  that  makes  you  indifferent  is  ? 


Table  1.  Final  versions  of  the  formats  for  eliciting  Indifference  proba¬ 
bilities  _  _ _ 


Format  Four 
A 
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I 

GPA 

I  IGPA 

I 

GPA 

I 

I 

I  I 

I 

I 

I 

2.5 

I  11.5 

I 

3.5 

I 

I 

I  I 

I 

I 

100Z 

20Z 

80Z 

OPTION 

0 

indifferent 

1 

FOR  SURE 

2 

CHANCE 

3 

RESTART 

Which  option  do 

you  prefer? 

B 

FOR  SURE  OPTION 

CHANCE  OPTION 

I 

CPA 

I  IGPA 

GPA 

I 

I 

I  I 

I 

I 

2.5 

I  11.5 

3.5 

I 

I 

I  I 

I 

100Z  iooz-p  p 


Your  Indifference  probability,  P,  has  been  determined  to  be 
between  40Z  and  60Z. 

Value  of  P?  _ 


i> 


Table  1  Final  versions  of  the  formate  for_  eliciting  indifference  probe 
con't.  bilitiee 
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Format  One  asked  subjects  to  directly  specify  the  value  of  p  that 
will  make  than  indifferent  with  respect  to  the  for-sure  and  chance  options. 
Hence  it  is  known  as  the  direct  probability  elicitation  format.  Originally 
subjects  were  asked  to  state  probability  values  but  in  the  final  version 
of  the  format  they  stated  percents  Instead.  He  found  that  subjects 

had  difficulty  thinking  in  terms  of  mathcMtlcal  probabilities  and  were  much 
more  comfortable  dealing  with  percents.  Appendix  A  shows  the  evolution 
of  this  and  other  formats  from  our  original  conceptions  of  them  to  their 
final  forms. 

Format  Two  differs  from  Format  One  in  that  subjects  were  asked  to 
state  preferences  between  the  two  options  given  certain  values  for  p. 

The  computer  asked  for  option  preference  or  indifference  regarding  proba¬ 
bilities  in  the  following  order:  .1,  .9,  .3,  .7,  .4,  .6,  .5  (alternatively, 
.9,  .1,  .7,  .3,  .  .  .).  If  the  indifference  probability  was  found  to  be 
between  .6  and  .7,  say,  then  the  questioning  procedure  continued  with  values 
of  .62,  .68,  .64,  .  .  .  (.68,  .62,  .66  .  .  .).  In  this  manner  format  two 
zeroed  in  on  the  subject's  indifference  probability  for  each  situation. 
Because  of  the  order  in  which  the  p  values  were  presented.  Format  Two  is 
known  as  the  ends-in  method. 

Format  Three,  the  quartering  method,  divided  the  probability  scale, 

.03  -  .93, into  approximate  quarters  andattempted  to  determine  in  which 
quarter  the  subject's  indifference  probability  lay.  Subjects  were  first 
asked  whether  the  probability  that  would  make  them  indifferent  between 
the  two  options  was  larger  or  smaller  than  .3.  Depending  on  their  answer 
to  that  first  option,  they  were  then  asked  whether  their  indifference 
probability  was  greater  or  smaller  than  either  .25  or  .75.  The  responses 
to  these  two  questions  placed  the  subject's  indifference  probability  in 
one  of  the  following  four  regions:  .05  -  .25,  .25  -  .SO,  .50  -  .75,  and 
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.75  -  .95.  Once  Che  appropriate  region  was  determined,  subjects  were  pre¬ 
sented  with  a  list  of  probabilities  at  .5  intervals  and  asked  to  choose 
the  set  of  probabilities  (each  set  Included  p  and  1  -  p)  that  was  closest 
to  asking  them  indifferent  between  the  two  options.  Values  for  p  below 
.5  were  presented  in  ascending  order  (e.g.,  .05,  .10,  .15,  .20,  .25)  and 

probabilities  above  .5  were  presented  in  descending  order  (e.g.,  .75, 

.70,  .65,  .60,  .55,  .50).  This  was  done  in  an  atteapt  to  avoid  an  anchor 
at  .5. 

The  modified  ends-in  method  with  graphic  representation.  Format  Four, 

was  designed  in  an  attempt  to  render  the  situations  easier  for  unsophisticated 
subjects  to  understand,  thus  making  it  easier  for  thaa  to  respond.  The 
field  in  which  the  chance  option  was  displayed  was  divided  into  nine  equal 
parts,  thus  allowing  any  of  the  probabilities  from  .1  to  .9  at  .1  inter¬ 
vals  to  be  graphically  portrayed.  As  in  the  original  ends-in  format, 
subjects  were  asked  to  state  their  preference  (or  indifference)  between 
the  for-sure  and  chance  options  as  the  values  of  p  changed.  Although  orig¬ 
inally  p  alternated  as  follows-  .1,  .9,  .2,  .8,  .3,  .7,  .4,  .6,  .5  (alter¬ 
natively,  .9,  .1,  .8,  .  .  .)-  this  procedure  was  found  to  be  tedious  and 
tiae-consuming.  2n  the  final  version  for  Experlaent  3  the  nuaber  of  p 
values  subjects  considered  was  drastically  reduced  (.2,  .8,  .4,  .6  or  ,8, 

.2,  .6,  .4).  If  the  Indifference  probability  was  found  to  be  between  .4 
and  .6  (.6  and  .7  in  the  original  version),  say,  the  subjects  were  asked 
to  directly  specify  probabilities  of  getting  the  higher  and  lower  GPA's 
in  the  chance  option  that  would  aake  thea  indifferent  between  the  two  options. 
Note  that  whereas  the  above  explanations  were  all  given  in  terns  of  proba¬ 
bilities,  Formats  Three  and  Four  and  the  final  version  of  Format  One  were 
actually  presented  to  the  subjects  in  terms  of  percents  (l.e.,  the  proba¬ 
bility  values  for  the  chance  options  were  actually  percents  adding  up  to  100Z). 
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The  aoit  coherent  gambles  (adjacent  and  distant  symmetric  gambles) 
and  formats  (Formats  One  and  Four)  were  taken  together  to  fora  the 
three  fixed  state  utility  assessasnt  procedurea  for  Experiaent  3.  Fifteen 
Indifference  probabilities  were  elicited  for  each  subject.  In  the  SFS 
procedure  a  nonlinear  least  squares  (LSQ)  fit  of  the  data  points  was  then  aade 
and  subjects  were  subsequently  allowed  to  re-assess  their  probabilities 
using  their  specified  set  as  the  working  set.  Subjects  were  peraltted  to 
re- specify  their  indifference  probabilities  a  aaxiaua  of  two  tlaes.  It 
has  been  our  experience  that  after  this  tine  few  significant  changes  are 
likely  to  be  aade.  If  subjects  did  not  wish  to  chsnge  their  specified 
probabilities  after  the  LSQ  fit  but  were  also  not  satisfied  with  the 
fitted  probabilities,  they  were  allowed  to  go  directly  to  the  questionnaire 
anyway.  Mo  subject  was  encouraged  to  change  his/her  Indifference 
probabilities,  only  to  think  about  the  situations  carefully.  In  this 
manner  as  much  coherence  as  possible  among  the  indifference  probabilities, 
and  thus  the  utilities,  was  obtained.  Novick  and  Llndley  (1978)  have 
stressed  the  Importance  of  obtaining  coherence. 
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In  the  RC  procedure  Indifference  probabilities  were  elicited  for 
two  gambles  using  either  Format  One  or  Format  Four.  The  subjects  were 
then  presented  with  a  table  showing  the  initial  situations  with  their  stated 
indifference  probabilities  and  two  additional  situations  with  computer-deduced 
indifference  probabilities.  They  were  told  that  their  initial  responses 
implied  certain  specific  indifference  probabilities  for  the  two  new  situations 
(situations  three  and  four  below).  The  following  table  illustrates  the 
the  latter  part  of  this  procedure. 


Situations 

2 

i 

p  chance 

1.50 

2.00 

2.00 

2.00 

for  sure 

1.00 

1.50 

1.00 

1.50 

lOOZ-p  chance 

0.50 

1.00 

0.50 

0.50 

p«53Z 

P-58Z 

P-40Z 

P-75Z 

Table  2.  RC  Procedure 

After  studying  the  table  the  subjects  were  given  an  opportunity  to 
change  the  Indifference  probabilities  for  two  situations  at  a  time  until 
they  expressed  indifference  with  respect  to  the  two  options  (chance  and 
for-sure)  in  each  of  the  four  situations.  In  this  way  subjects  were  helped 
to  make  their  probability  judgments  coherent  for  a  region  of  grade  point 
averages.  When  the  subjects  were  satisfied  with  all  four  indifference 
probabilities,  the  procedure  began  again  with  specification  of  indifference 
probabilities  for  two  new  situations.  A  LSQ  fit  of  the  data  points  was 
done  at  the  end  of  the  RC  procedure  but  was  not  shown  to  the  subject.  This 
allowed  us  to  see  how  coherent  the  entire  set  of  indifference  probabilities 
was.  Although  the  RC  procedure  determines  that  each  group  of  four  Indifference 
probabilities  is  internally  consistent,  it  does  not  guarantee  that  the 
probabilities  will  be  consistent  across  groups. 
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The  LC  procedure  presented  subjects  with  two  types  of  hypothetical 
choice  situations:  (1)  a  for-sure  and  a  chance  option  (the  standard  gamble 
used  in  the  SFS  and  RC  procedures)  and  (2)  tiro  chance  options.  Again, 
either  Format  One  or  Format  Four  was  used  to  elicit  an  indifference 
probability  for  the  first  situation,  after  which  the  subject  was  told  that 
that  response  implied  that  he/she  should  be  Indifferent  with  respect  to  the 
two  options  in  situation  2  for  the  designated  probabilities.  (The  Option  1 
probabilities  for  Situation  2  were  always  50Z/50Z  for  the  two  highest  GPAe  as 
shown.  Only  the  Option  2  probabilities  for  the  highest  and  lowest  GPAs  changed.) 


GAMBLE  SITUATION  1 


GAMBLE  SITUATION  2 


I 

I 

I 

I 

I 


OPTION  1 
(FOR  SURE) 

0.50 


1 


OPTION  2  * 

(CHANCE)  * 
1.00  50Z  J 

0.00  50Z  1 


*  OPTION  1  OPTION  2 

l  50Z  1.00  75Z 

J  50Z  0.50  — 

1  —  0.00  25Z 


Table  3.  LC  Procedure 

If  the  subject  was  not  indifferent  between  the  options  in  both  situations, 
he/she  was  allowed  to  modify  the  Situation  1  indifference  probability. 

After  that  probability  was  changed  the  subject  was  again  presented  with  a 
table  similar  to  Table  3  above,  with  the  Situation  2  Option  2  probabilities 
now  also  different.  This  continued  until  the  subject  was  indifferent  with 
respect  to  the  options  in  each  of  the  situations.  At  that  point  the  entire 
procedure  was  repeated  with  a  new  set  of  GPAs  (i.e.,  a  new  gamble).  As 
with  the  RC  procedure,  a  LSQ  fit  of  the  Indifference  probabilities  was 

computed  after  the  subject  had  left. 

An  important  feature  of  these  experiments  was  that  the  primary 

interaction  was  between  the  assessor  and  a  preprogrammed  computer 
presentation  of  the  experimental  protocol.  This  presumably  led  to  a 
high  degree  of  uniformity  in  presentation.  However,  research  assistants  did 
read  text  and  answer  questions  so  it  was  possible  that  soma  inter- 
assistant  variability  might  be  present. 
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Pilot  Experiment  1  -  Formats  and  gaablas  for  tha  SFS  procadura 

Experiaent  1  was  designed  to  deteraine  which  of  the  first  two  for¬ 
mats,  direct  probability  elicitation  and  ends-in,  for  presenting  gaablas  in 
the  three  fixed  state  assessaent  procedures  elicited  the  aost  coherent, 
bias-free  set  of  indifference  probabilities.  The  experiaent  was  done 
in  the  context  of  the  SFS  procedure  and  the  results  ware  than  extrapo¬ 
lated  to  the  RC  and  LC  procedures.  Experiaent  1  also  determined 
which  gambles  resulted  in  the  largest  errors,  in  terns  of  log-odds 
differences,  and  thus  should  be  oaltted  from  the  utility  assessaent 
procedures . 

Method 


Subjects.  The  subjects  for  the  first  part  of  experiaent  1  were 
43  college  undergraduates  of  both  sexes  who  responded  to  newspaper  and 
other  advert iseaents.  The  students  were  randoaly  assigned  to  four  groups 
so  that  each  group  contained  approxiaately  the  same  nuaber  of  subjects 
of  each  sex.  Three  subjects  were  excluded  from  the  analyses  because  they 
failed  to  get  through  the  entire  prograa  and  thus  contributed  no  data  to 
the  file.  All  subjects  were  paid  $4.00  for  their  participation  in  this 
experiment. 

The  subjects  for  the  second  phase  of  this  experiment  were  PhD  level 
professionals  in  the  fields  of  Education,  Psychology,  Measureaent,  and 
Computer  Science,  and  a  few  advanced  graduate  students, attending  the 
Advanced  Seainar  in  Bayesian  Statistics  supported  by  the  CADA  Research 
Group  at  The  University  of  Iowa.  These  subjects  served  as  volunteers. 

Materials  and  Design.  For  all  experiments  Cbmponent  31  of  the 
Computer-Assisted  Data  Analysis  (CADA)  Monitor  (Novlck,  et  al. ,  1980) 
was  modified  slightly  and  then  used  in  a  conversational  mode  to  present 
the  formats,  gaables,  and  Incoherence  resolution 

methods  associated  with  each  assessaent  procedure.  The  Monitor  was 
accessed  via  a  cathode  ray  tube  (CRT)  terminal  linked  to  a  PDF  11V03 
coaputer.  Individual  sessions  were  conducted  by  the  research  assistants 
who  operated  the  coaputer  systea,  recording  verbal  responses  given  by 
each  participant.  A  complete  recording  of  the  protocol  of  each  session 
was  stored  on  flexible  disk  and  made  available  in  hard  copy  through  a 
Decwriter  terminal.  Data  analyses  were  performed  on  the  11V03  using 
data  stored  on  the  flexible  disks. 


Th«  following  two  sets  of  gasiblss  were  used  alternately  with  formats 
one  end  two:  11]— (0.0,  0.5,  1.0),  (0.5,  1.0,  1.5),  (0.0,  1.5,  3.0), 

(0.0,  1.5,  4.0),  (1.5,  2.0,  2.5),  (1.0,  2.0,  3.0),  (0.0,  2.5,  4.0), 

(2.0,  3.0,  4.0),  (0.0,  3.5,  4.0)  end  [2]— (0.0,  0.5,  4.0).  (0.0,  1.0, 

2.0),  (0.0,  1.0,  4.0),  (1.0,  1.5,  2.0),  (0.5,  2.0,  3.5),  (2.0,  2.5,  3.0), 
(1.5,  2.5,  3.5),  (0.0,  3.0,  4.0),  (3.0,  3.5,  4.0).  For  the  professional- 
level  subjects,  average  GRE  scores  rather  than  GPAs  were  used.  To  con¬ 
struct  the  gamble  sets,  400  was  substituted  for  0.0,  450  for  0.5,  etc. 
up  to  800  for  4.0.  The  nine  gambles  comprising  each  set  were  chosen  so 
as  to  make  the  two  sets  of  gambles  as  similar  as  possible.  Even  so, 
both  formats  were  used  with  each  set  of  gambles.  The  subjects  in  group 
one  responded  to  the  first  set  of  mixed  gambles  using  Format  One  and 
then  the  second  set  using  Format  Two.  Group  two  subjects  responded  first 
to  set  one  using  Format  Two  and  second  to  set  two  using  Format  One. 

Groups  three  and  four  responded  to  the  set  two  gambles  first,  using  For¬ 
mats  One  and  Two  respectively.  The  design  was  thus  a  crossed  formats 
by  gambles  design  with  a  further  administration  of  the  alternate  format- 
gamble  pair  to  each  group.  A  resolution  program  was  used  to  determine 
the  best  format,  in  the  sense  of  minimum  mean  squared  deviations  from  a 
least  squares  fit  over  all  gambles  made  by  the  participant.  (See  data 
analysis  section.)  This  design  thus  permitted  examination  of  the  for¬ 
mat  effect  with  a  check  and  adjustamnt  for  gamble-set  and  order  effects. 

Procedure.  The  experiment  was  conducted  in  a  private  office  with 
subjects  being  tested  individually  in  one-hour  sessions.  Each  subject 
participated  in  one  session.  All  instructions  were  standardised  and  printed 
on  a  CRT  terminal.  The  research  assistant  read  the  instructions  to  the 
undergraduates  to  insure  that  they  dearly  understood  what  mis  expected 
°*  them.  The  professional-level  subjects,  however, were  permitted  to  read 
the  instructions  on  their  own,  with  the  research  assistant  there  to 
answer  any  questions  they  may  have  had.  Subjects  were  given  very  careful 
instructions  in  the  meaning  of  indifference  probabilities  and 
in  the.  assessment  of  them.  The  concept  of-  utility 

was  never  mentioned,  nor  were  a  subject's  utilities  ever  displayed,  because 
of  the  difficulty  of  making  utility  theory  understandable  to  people  with 
little  or  no  statistical  background.  Some  of  the  professional-level  subjects 
were  familiar  with  Bayesian  statistics  and  utility  theory  and  this  was  duly 
recorded.  Except  for  the  exact  sets  of  Instructions  printed,  all  of  the 
above  was  standard  procedure  in  both  of  the  pildt  experiments. 

In  addition  to  the  general  instructions,  specialized  instruc¬ 
tions  were  given  for  each  elicitation  format.  Prior  to  indicating  their 

indifference  probability,  subjects  were  given  a  chance  to  review 
the  instructions  to  insure  that  they  clearly  understood  what  they  were 
to  do.  After  the  introductory  paragraph  and  before  the  instructions 
concerning  the  nature  of  the  choice  situations,  the  following  four  ques¬ 
tions  were  asked  of  undergraduates  to  get  them  to  begin  considering  how 
their  final  grade  point  average  would  affect  their  future  opportunities: 
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1.  WHAT  YEAR  IN  SCHUUL  ARE  YOU!* 

2.  WHAT  IS  YOUR  MAJORS 


J.  UPON  COMPLETION  OF  YOUR  BACHELOR'S  DEGREE *  WHICH  OF  THE  FOLLOWING 
POSSIBILITIES  DO  YOU  CONSIDER  MOSI  LIKELY  TO  OCCUR? 

GRADUATE:  STUDY  TO  HAS  I  IK'S  DEGREE  l  LULL 

GRADUATE  STUDY  10  PHD  l  LULL  OR  PROFESSIONAL  SCHOOL 

EMPLOYMENT 

Hi  HER 

ANSWER ? 

4.  10  WHAT  EXTENT  DO  YOU  FELL.  YOUR  CUMULATIVE  GPA  WILL  AFFECT  WHAT  YOU  WANT  TO 

DU  IN  IHt  FUTURE7 


GREATLY 
SOMEWHAT 
VERY  LITHE 
ANSWER? 


In  order  to  get  the  professional-level  subjects  to  begin  thinking 
about  the  importance  of  GRE  scores  in  selecting  first-year  graduate 
students,  the  following  four  questions  were  asked  of  them: 


1.  WHAT  IS  THE  ACADEMIC  DISCIPLINE  FOR  WHICH  YOU  MIGHT  SERVE  AS  CHAIR  OF 
GRADUATE  ADMISSIONS? 

2.  WHAT  IS  THE  APPROXIMATE  MEAN  AVERAGE  GRE  SCORE  OF  FIRST-YEAR  GRADUATE 
STUDENTS  IN  THE  DEPARTMENT  IN  WHICH  YOU  MIGHT  SERVE?  RECALL  THAT  EACH 
SUBTEST  OF  THE  GRE  IS  ON  THE  SCALE  200-900  WITH  A  NATIONAL  MEAN  OF  ABOUT 
500  AND  STANDARD  DEVIATION  OF  ABOUT  100, 

ANSWER? 

3.  DO  YOU  FEEL  THAT  THIS  SCORE  IS  A  GOOD  PREDICTOR  OF  FUTURE  PERFORMANCE 
IN  YOUR  PROGRAM? 

1.  YES*  VERY  GOOD  PREDICTOR 

2.  YES*  MODERATELY  GOOD  PREDICTOR 

3.  AVERAGE  PREDICTOR 

4.  NOf  FAIR  PREDICTOR 

5.  NO*  POOR  PREDICTOR 

ANSWER? 

4.  DO  YOU  FEEL  THAT  THE  TEST  SCORE  IS  AN  IMPORTANT  PIECE  OF  INFORMATION  IN 
SELECTING  STUDENTS? 

1.  YES*  VERY  IMPORTANT 

2.  YES*  IMPORTANT 

3.  NO  STRONG  FEELINGS  EITHER  WAY 

4.  NO*  NOT  VERY  IMPORTANT 
3,  NO*  NOT  IMPORTANT  AT  ALL 


ANSWER? 
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A  complete  sec  of  the  instructions  given  to  these  subjects  uy  be  found 
in  Appendix  B .  Except  for  s  few  slight  aodificstions  to  neks  the  instruc¬ 
tions  eesier  to  understsnd,  the  sene  instructions  were  used  for  the  pro- 
fessionel-level  subjects  ss  for  the  undergreduetes.  Thus  to  derive  the 
instructions  for  the  undergreduetes,  one  csn  sieply  substitute  "final 
grsde  point  everege  st  greduetion"  for  "admittance  of  e  student  with  e 
psrticuler  everege  GKZ  score. M  The  psregrephs  introducing  the  scenerlo 
were  different  for  both  sets  of  subjects  end  thus  both  introductions 
are  reprinted  in  Appendix  1 . 

After  providing  indifference  probabilities  for  both  sets  of  gambles, 
subjects  were  shown  a  least  squares  fit  of  the  data  points.  They  were 
told  that  the  coaputer  had  generated  a  set  of  Indifference  probabilities 
similar  to  theirs  in  order  to  help  then  specify  a  coherent  set  of  p 
values.  If  subjects  were  not  satisfied  with  the  fitted  probabilities, 
they  were  given  an  opportunity  to  change  any  or  all  of  their  specified 
probabilities.  A  new  least  squares  fit  based  on  the  revised  probabili¬ 
ties  was  then  coaputed.  Subjects  were  allowed  to  revlae  their  indiffer¬ 
ence  probabilities  twice  if  they  so  desired  and  thus  were  able  to  see 
up  to  three  different  sets  of  fitted  probabilities.  Any  subject  who  had 
not  accepted  the  fitted  probabilities  by  the  third  tine  was  asked  to  do 
so  then  because  it  has  been  our  experience  that  by  then  neither  set  of 
probabilities  is  likely  to  change  substantially.  When  the  subject  ac¬ 
cepted  the  fitted  probabilities  the  fornal  part  of  the  experiment  was 
coaplete. 


At  the  end  of  the  experiment  all  subjects  responded  to  a  questionnaire 
calling  for  an  evaluation  of  the  experiment  as  a  whole.  If  subjects 
inquired  further  about  the  purpose  of  the  experiment  they  were  told  that 
the  results  would  be  used  to  help  design  procedures  for  making  it  easy 
for  persons  to  decide  on  desirable  gaables. 
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Fllot  IxptrlMnt  2  -  Formats  for  standard  flxad  atata  gambles 

Experiment  2  was  an  axtanalon  of  experiment  1;  it 'a  purpose  was  to 
determine  which  of  two  new  formats  (formats  3  and  4)  for  eliciting 
Indifference  probabilities  provided  the  most  coherent,  bias-free  set  of 
probabilities.  A  revised  set  of  gambles  was  used. 

Method 

Subjects.  The  10  subjects  were  graduate  students  enrolled  in  the 
Bayesian  Statistics  1  class  at  The  University  of  Iowa.  They  served  as 
volunteers. 

Design.  The  design  was  a  crossed  formats  by  gambles  design  with  a 
further  administration  of  the  alternate  format-gamble  pair  to  each  gToup 
as  in  experiment  1.  Gamble  set  one  consisted  of  the  following  adjacent 
gambles:  (0.5,  1.0,  1.5),  (1.0,  1.5,  2.0),  (1.5,  2.0,  2.5),  (2.0,  2.5, 
3.0),  and  (2.5,  3.0,  3.5).  Gamble  set  two  included  the  following  distant 
symmetric  (specifically,  two-apart)  gambles:  (0.0,  1,0,  2.0),  (0.5, 

1.5,  2.5),  (1.0,  2.0,  3.0),  (1.5,  2.5,  3.5),  and  (2.0,  3.0,  4.0).  In 
addition,  whichever  set  of  gambles  was  responded  to  first,  regardless 
of  format,  also  included  (0.0,  0.5,  1.0)  and  (3.0,  3.5,  4.0)  so  that 
the  first  seven  situations  were  sufficient  to  determine  a  utility 
function.  The  second  set  of  gambles  to  which  a  subject  responded  inclu¬ 
ded  (0.0,  2.0,  4.0)  and  (1.0,  2.5,  4.0).  Thus  each  subject  indicated 
Indifference  probabilities  for  a  total  of  14  situations.  Computation 
of  format  errors  did  not  consider  the  above  four  gambles.  These  14  gambles 
were  chosen  on  the  basis  of  the  results  of  Pilot  Experiment  1.  The 
utility  gambles  were  excluded  because  subjects  found  them  difficult 
and  because  they  generally  resulted  in  large  log-odds  differences  between 
the  assessed  and  fitted  indifference  probabilities.  The  four  experi¬ 
mental  groups  were  constructed  as  in  the  previous  experiment. 

Procedure.  The  procedure  was  the  same  as  in  experiment  1  except 
that  each  subject  indicated  only  14  Indifference  probabilities  rather 
than  18. 
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Experiment  3  -  Comparison  of  SFS,  RC,  and  LC  procedures 

The  purpose  of  this  experiment  was  twofold.  Pirst  we  were  interested  in 
whether  the  original  forest  (Format  One)  or  a  new  format  (Forest  Four) 
resulted  in  the  aost  reliable  (coherent)  and  bias-free  set  of  utilities 
(indifference  probabilities).  The  aajor  conclusion  of  interest,  however, 
was  which  of  the  three  utility  assessment  procedures  provided  the  best 
set  of  utilities. 

Method 

Subjects.  Although  SO  subjects  were  run,  10  had  to  be  excluded  from 
the  data  analyses  because  a)  they  failed  to  get  through  the  entire 
procedure  or  b)  their  indifference  probabilities  were  too  Incoherent  for 
a  least  squares  fit  to  be  computed.  The  remaining  40  were  randomly  assigned 
to  three  groups,  with  approximately  the  same  number  of  subjects  in  each 
group.  Subjects  were  paid  $4.00  upon  completion  of  the  experimental 
session.  Students  who  had  participated  in  either  of  the  previous  pilot 
experiments  were  not  allowed  to  be  in  this  experiment. 

Design.  In  this  experiment  format  was  a  withln-subject  variable 
while  assessment  procedure  and  format  order  were  between-subject  variables. 
Each  group  of  subjects  used  only  one  of  the  three  procedures.  Half  of 
the  subjects  in  each  group  used  Format  One  followed  by  Format  Four,  while 
the  other  subjects  used  Format  Four  first.  There  were  three  or  four 
subjects  of  each  sex  in  each  format-order  group  for  each  procedure. 

Format  One,  direct  elicitation,  was  chosen  because  it  is  the  format 
most  often  used  with  utility  assessment  procedures.  Format  Four,  on  the 
other  hand,  was  chosen  because  Pilot  Experiments  1  and  2  suggested  that 
it  may  be  better  than  Format  One,  in  the  sense  of  being  easier  for 
assessors  to  use,  in  inducing  more  coherent  responses  from  them,  and  in 
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seemingly  being  less  influenced  by  che  anchoring  effect.  The  gambles  were 
chosen  on  the  basis  of  the  same  information  as  led  to  the  format  choices  and* 
in  general,  were  the  ones  subjects  found  easiest  to  think  about.  The  only 
utility  gamble  used  is  (0.0,  2.0,  4.0)  because  subjects  generally  found 
it  very  difficult  to  specify  indifference  probabilities  for  utility  gambles. 
As  a  group,  the  utility  gambles  led  to  much  more  incoherent  judgments 
than  either  the  adjacent  or  distant  symmetric  gambles.  There  were  a 
few  other  gambles  (e.g.  (0.0,  0.5,  1.0)  and  (3.0,  3.5,  4.0))whlch  were 
a  little  difficult  for  subjects  but  these  were  retained  because  they 
were  needed  to  compute  the  utility  function.  Also,  these  gambles  were  not 
as  bad  as  the  utility  gambles. 

Two  mixed  seta  of  gambles  (containing  both  adjacent  and  distant 
symmetric  gabbles)  were  constructed.  The  seven  gambles  in  Set  One  were 
sufficient  to  compute  a  utility  function  and  were  used  as  the  basis  for 
determining  the  fitted  probabilities  for  the  SFS  procedure.  Set  Two 
Included  all  of  the  remaining  adjacent  and  distant  gambles.  The  gamble 
sets  were: 

SET  OHE  -  (0.0,  0.5,  1.0),  (0.0,  1.0,  2.0),  (0.5,  1.5,  2.5),  (1.5,  2.0, 
2.5),  (1.0,  2.5,  4.0),  (2.5,  3.0,  3.5),  and  (3.0,  3.5,  4.0). 

SET  TWO  -  (0.5,  1.0,  1.5),  (1.0,  1.5,  2.0),  (0.0,  1.5,  3.0),  (1.0,  2.0, 
3.0),  (0.0,  2.0,  4.0),  (2.0,  2.5,  3.0),  (1.5,  2.5,  3.5),  and  (2.0,  3.0,  4.0). 
These  two  sets  were  used  as  presented  above  for  both  the  SFS  and  LC 
procedures.  Because  of  the  manner  in  which  situations  three  and  four  were 
generated  in  the  RC  procedure,  it  was  necessary  to  substitute  (0.0,  1.0,  2.0) 
and  (2.5,  3.0,  3.5)  for  (0.0,  1.5,  3.0)  and  (2.0,  2.5,  3.0)  in  Set  Two 
for  that  procedure.  Thus  subjects  responded  to  these  two  gambles  twice 
in  the  RC  procedure  but  only  once  in  the  other  two  procedures. 
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Subjects  alifays  responded  to  Che  Sec  One  gaables  before  proceeding 
to  the  Sec  Two  gaables.  The  forests ,  however,  as  explained  above,  were 
oouncerbalanced  wich  half  of  Che  subjeccs  using  each  procedure  receiving 
Foraat  One- Set  One/Foraat  Four-Sec  Two  and  Che  ocher  half  receiving  ForaaC 
Four-Sec  One/Foraac  One-Sec  Two.  This  was  done  in  order  co  enable  us 
Co  look  ac  foraat  effects  wichln  and  across  procedures. 

The  design  was  chus  a  crossed  procedure  by  sex  by  forast-order  design. 
Caablc  sec  order,  however,  remained  fixed. 

Procedure.  The  procedure  for  Chis  experiaent  was  in  general  Che  saae 
as  Chat  for  chc  two  piloC  experiments.  There  were,  however,  a  few 
differences.  First,  the  exacc  procedure  followed  by  each  subject  was 
obviously  dependent  on  whether  he/she  used  Che  SFS,  RC,  or  LC  procedure. 

In  the  previous  studies  all  subjects  used  the  SFS  procedure.  Also,  in 
this  experiaent  we  had  subjects  write  down  their  utility  function  for 
grade  point  averages  (from  0.0  to  4.0)  on  a  scale  of  0-100  after  they 
had  coapleted  the  probability  elicitation  procedure.  This  question  was 
in  addition  to  the  general  written  questionnaire. 
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Data  Analysis,  ideally  a  data  analysis  would  conpare  assessed  utilities  with 
true  utilities.  Unfortunately  there  is  no  assured  method  of  measuring 

true  utilities.  However,  an  approach  commonly  adopted  in  psychological 
testing  seems  relevant.  Assuming  that  assessed  utilities  involving 
extensive  measurements  with  averaging  and/or  adjustment  are  likely  to 
be  close  to  true  utilities,  the  reliability  or  accuracy  of  individual 
component  procedures  can  be  assessed  by  comparing  utilities  from  these 
procedures  with  those  from  the  more  extensive  measurement.  The  testing 
analogy  would  be  to  evaluate  each  of  several  short  measures  of  a 
construct  by  relating  each  measure  to  the  average  overall  measures  or 
perhaps  to  some  factor  analytically  defined  composite. 

A  variety  of  nethods  of  comparing  measures  is  possible.  He  have 
chosen  to  use  an  approach  that  is  closely  related  to  the  least  squares 
fitting  procedure  (Novick  &  Lindley,  1979).  Consider,  for  example, 
the  comparison  of  two  methods  (I  and  II),  each  Involving  a  fixed 
state  assessments  for  N  points,  with  coherence  adjustment.  Denote  the 
N  utilities  obtained  for  the  two  methods  of  Uj.^,  ...,  U^N  and 
...,  UIIN.  Denote  the  N  assessed  utilities  after  the  least  squares  fit 
and  resolution  involving  the  2m  gambles  by  U^,  . ...  U^.  The  symbols 
Un,  and  y  will  be  used  to  denote  vectors  containing  these  ordered 
utilities.  For  each  gamble  used  there  will  be  a  probability  Implied 
by  each  of  the  vectors  Bj,  Ujj.  end  y.  He  denote  these  as  value 

...»  *1b|,  •••»  *n<1  •••*  *nd  the  corresponding 

vectors  by  »I1,  and  v.  Then  taking  the  log  odds  (y  ■  log  [*/(!-*))) 
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of  the  individual  probabilities  ve  can  evaluate  each  method  by 
considering  discrepancies  between  transformed  probabilities  for 
the  Individual  methods  and  the  corresponding  values  for  the  final 
fitted  utilities.  In  these  analyses  we  shall  take  the  values  y^ 

....  yN  for  the  resolved  log-odds  probabilities  as  "true"  values 
and  consider  values  . and  ••••  *s  measure¬ 

ments  containing  independent,  normal,  homoscedastic  errors.  The  symbols 
y,  ,  and  will  denote  the  vectors  of  log-odds  values.  Data 
analysis  will  then  Involve  the  comparison  of  the  average  bias  and 
average  squared  error  for  the  methods  in  question.  Symbolically  the 
comparisons  are  (YI1“Y1>.  (yi2-Y2),  •••,  (yih-Yh)  end  (Yjj^-Yj) , 
(y1i2-Y2) »  • • • •  <Yiin-Yn) •  In  the  several  experiments  this  involves 
comparing  various  categories  of  method,  format,  type  of  gemble,  and 
individual  gambles. 

The  normality  and  homoscedasticity  assumptions  are  reasonable 
on  theoretical  grounds  by  vlrture  of  the  log-odds  transformation  and 
are  supported  by  data  exploration.  Slightly  less  comforting  is  the 
assumption  that  errors  are  independent  (Novlck  &  Lind ley,  1979). 

However,  measurement  Independence  is  a  standard  casual  assumption  that 
seems  to  be  as  acceptable  here  as  in  other  applications.  Failure  of 
this  assumption  would  be  important  primarily  if  it  were  differential 
between  methods  being  compared. 
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Results 

Two  dependent  variables.  Mean  Absolute  Error  (MAE)  and  Greatest 
Absolute  Difference  in  Utilities  (GADU)  were  considered  as  criteria 
of  subjects'  performance.  These  measures  and  the  relationship  between 
them  will  be  discussed  in  detail  below.  The  tables  presenting  the 
results  to  be  discussed  are  located  in  Appendix  D. 

Analysis  with  MAE  as  the  Dependent  Variable 

The  primary  criterion  of  a  subject's  performance  was  Mean  Absolute 
Error,  an  index  of  overall  consistency.  Three  types  of  MAE  measures 
were  utilized  in  the  analysis.  The  overall  measure  for  each  subject 
was  obtained  by  averaging  the  15  absolute  log-odds  differences  between 
his/her  specified  probabilities  and  the  probabilities  obtained  from  the 
least-squares  fit  (see  Table  4).  The  distribution  of  MAE  was  determined 
to  be  approximately  normal  by  CADA's  Normal  Probability  Plot.  The 
grand  mean  and  the  standard  deviation  for  the  measure  were  .4065  and 
.1800,  respectively,  with  observations  ranging  from  .0555  (very  consistent) 
to  .8540  (very  inconsistent). 

Because  of  the  interest  in  performance  under  each  of  the  two  formats, 
two  additional  MAEs  were  computed  for  each  subject:  one  for  performance 
using  Format  1  and  one  for  performance  using  Format  4.  The  cell  means 
and  n-counts  for  the  analysis  of  the  effects  of  Procedure,  Format, 
and  Sex  are  presented  in  Table  5. 

A  final  measure  of  MAE  was  considered  appropriate  for  some  aspects  of 
the  analysis.  This  variable  may  be  described  as  the  difference  between 
the  MAE  obtained  under  Format  1  and  the  MAE  obtained  under  Format  4 
(A  MAEj_^).  It  was  hypothesized  that  MAE^  would  be  somewhat  greater  than 
MAE^,  and  so  a  small  positive  value  was  expected  for  each  Individual. 
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Aa  can  be  aeen  from  Tabic  6,  the  grand  aean  for  AMAEj^  vac  slightly 
poaitive  (M  ■  .0046),  although  there  were  many  exceptiona  for  the 
Individual  obeervatlona  and  alao  for  aeveral  of  the  cell  means. 

A  two-way  ANOVA  waa  applied  to  the  new  variable  AMAEj_^  in  order 
to  determine  the  extent  to  which  it  waa  affected  by  Procedure  and  Sex. 
Table  6  shows  the  marked  Procedure  effect  upon  AMAEj_^  (39Z  of  the  total 
variance  in  AMAE^ can  be  accounted  for  by  Procedure).  In  particular, 
the  average  AMA£j_^  for  the  RC  procedure  was  relatively  high  (.0587), 
indicating  a  high  degree  of  interaction  between  this  procedure  and  the 
formats.  The  LC  and  SFS  procedures  evidenced  lesser  Interactions  with  the 
formats  (average  AMAE^_^  for  SFS  *  -.0242,  and  average  AMAEj_^  for 
LC  ■  -.0185).  These  results  are  discussed  in  more  detail  below. 

Main  Effects  for  MAE 
Procedure 

While  it  was  difficult  to  have  prior  convictions  about  the 
magnitude  of  the  Procedure  effects, the  authors  did  have  preconceptions 
about  the  efficacy  of  the  three  assessment  procedures.  In  particular, 
it  was  expected  that  the  Regional  Coherence  Procedure  would  be  most 
successful  in  eliciting  consistent  responses,  that  the  Standard  Fixed 
State  Procedure  would  be  least  effective,  and  that  the  Local  Coherence 
Procedure  would  be  intermediate.  However,  as  seen  in  Table 4, 
these  expectations  were  only  partly  realized.  The  MAE  for  RC,  SFS, 
and  LC  were  .3923,  .4021, and  .4295  ,  respectively.  The  poor 
performance  with  the  LC  procedure  was  thought  to  be  due  to  the  lack 
of  sophistication  of  the  undergraduate  subjects  and  might  not  be 
replicated  with  a  sample  of  subjects  who  had  had  more  statistical 
training.  A  procedure  for  making  the  LC  procedure  more  useful  might 
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consist  of  1)  revising  the  presentation  to  aake  it  more  comprehensible 
to  neophytes,  2)  rewriting  instructions  to  make  its  use  more  attractive, 
or  3)  using  it  with  only  more  sophisticated  subjects. 

Format 

The  effect  of  Format  on  MAE  was  surprisingly  small  (see  Table  S). 

We  had  expected  that  Format  4  would  yield  more  consistent  results.  Format 
may  be  considered  a  within  subject  factor  since  each  subject  was  run 
under  botn  formats,  and  so  40  observations  were  obtained  for  each  of 
of  the  two  formats.  The  average  MAE  under  Format  4  was  .4040,  which 
could  not  be  considered  to  be  appreciably  different  from  the  MAE  of  .4103 
obtained  under  Format  1.  The  Format  factor  might,  however,  be  considered 
interesting  due  to  its  interaction  with  Procedure  (to  be  discussed  below). 

Sex  of  Subject 

The  subjects'  sex  was  found  to  have  a  strong  effect 
on  performance  as  measured  by  MAE.  The  greater  consistency  on 
the  part  of  the  female  subjects  was  Interesting  for 
several  reasons.  First  of  all,  it  was  expected  that  males  would 
have  a  more  extensive  background  in  quantitative  disciplines  and 
might  therefore  excel  in  tasks  requiring  complex  reasoning  abilities. 
Secondly,  there  was  a  problem  with  students  who  had  made  appointments 
for  the  experiment  failing  to  report,  and  this  "no-show”  tendency 
was  markedly  more  prevalent  in  male  subjects.  Therefore,  it  was 
expected  that  the  males  who  did  keep  their  appointments  would  be 
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■or#  highly  motivated  and  more  likely  to  perform  well.  While  neither 
of  these  hypotheses  was  borne  out  by  the  MAE  data,  alternative 
explanations  for  the  greater  female  consistency  might  be  offered. 

In  particular,  the  results  could  be  seen  as  consistent  with  the 
general  finding  that  women  are  more  careful  than  men  on  many  tasks. 

They  might,  therefore,  be  expected  to  perform  better  In  an  experlamnt 
which  requires  great  deliveration  and  painstaking  accuracy. 

Interaction  Effects 

Procedure  by  Format  Interaction 

As  discussed  above.  Procedure  had  a  strong  effect  on  the  variable 
AMAEj_^ ,  accounting  for  39X  of  Its  variance.  A  complex  Interaction 
between  Foraat  and  Procedure  was  particularly  apparent  under  the 
RC  procedure  (see  Table  5). 

Since  one  of  the  objectives  of  the  experiment  was  to  determine 
which  combination  of  Procedure  and  Format  would  yield  the  most 
consistent  results,  particular  attention  should  be  paid  to  the 
average  MAEs  presented  in  Table  5.  it  should  be  noted  that  the 
RC  procedure  used  in  conjunction  with  Format  4  yielded  the  lowest 
average  MAE  (.3615).  This  finding 

confirmed  expectations  of  both  the  relative  efficacy  of  the  RC 
procedure  and  the  relative  ease  of  Format  4.  It  might  also  be 
noted  that  this  winning  combination  was  particularly  effective  for 
the  6  female  subjects  who  achieved  very  consistent  results  (x  ■  .2846) 
through  its  use.  On  the  other  hand,  the  LC  procedure  coupled  with 


-29- 


ForMC  4  produced  the  least  desirable  results  (x  -  .4387),  although 
these  results  did  not  differ  greatly  from  those  obtained  under  the 
LC  procedure  in  conjunction  with  Format  1  (x  *  .4202). 

Procedure  by  Sex  Interaction 

Table  4  deaonstrates  that  sales  performed  best  under  the  LC 
procedure  while  female  performance  was  worst  under  this  procedure. 
The  females  apparently  profited  from  the  RC  and  SFS  procedures 
(achieved  means  of  .2978  and  .3285,  respectively)  while  the  males 
did  rather  poorly  under  thaae  two  procedures  (with  mean  performance 
of  .4803  and  .5046,  respectively). 

Format  by  Sex  Interaction 

Table  5  indicates  little  Interaction  between  Format  and  Sex. 
Males  performed  only  soamwhat  better  under  Format  4  than  under 
Format  1  while  there  was  essentially  no  difference  in  female 
performance  as  a  function  of  format. 

Analysis  with  GAPU  as  the  Dependent  Variable 

A  second  criterion  of  performance  was  the  Greatest  Absolute 
Difference  in  Utilities  (GADU)  which  may  be  described  as  an  index  of 
congruence  between  the  subject's  stated  utility  function  (written  on 
a  piece  of  paper  before  or  after  the  experiment)  and  the  utility 
function  obtained  by  his  specification  of  probabilities  during  the 
procedure.  It  was  believed  that  this  night  be  a  better  indicator  of 
how  well  the  subject  was  abla  to  usa  the  procedure  to  reproduce  his 
GPA  gambling  behavior  than  would  the  consistency  index  (MAS) 
described  above.  This  assumption  was  based  on  the  observetlon  of 
cases  in  which  subjects  appeared  to  quite  consistent  (perhaps  merely 
by  specifying  .5  or  some  other  value  for  most  of  the  situations),  but 
had  fitted  utility  functions  which  deviated  greatly  from  their 


stated  functions. 


The  hypothesis  that  consistency  has  little  or  no  relationship 
to  success  in  reproducing  utility  functions  was  confined  by  obtaining 
a  Spearman  Rank-Order  Correlation  coefficient  for  the  ranks  of  each 
of  40  observations  on  overall  MAE  and  GABO.  The  r  of  -.0858  points  to  the 
fact  that  there  is  essentially  no  correlation  between  these  two  skills 
and  also  helps  to  explain  the  reversals  in  results  when  observations 
for  these  two  variables  were  broken  down  by  Procedure,  Sex, and 

Format  (see  analysis  for  MAE  above  and  GADU  below).  This 

lack  of  correlation  nay  cast  doubt  on  one  or  both  of  the  Indices  as 

Measures  of  successful  completion  of  the  gambling  task  or  it  may  be 

a  function  of  the  rather  small  undargraduate  sample  tested.  On 

the  other  hand,  the  essentially  sero  correlation  may  indicate  that 

both  indices  are  valid  measures  of  performance  but  that  they  identify 

very  different  skills. 

For  the  purposes  of  this  analysis,  the  40  subjects  were  divided 
into  two  groups  depending  on  the  format  they  used  first  in  the  utility 
assessment  procedure.  This  was  done  for  the  following  reason:  The 
15  gaafcle*  used  in  this  experiment  were  divided  into  two  sets.  Set  1, 
containing  7  gambles,  was  always  responded  to  first  while  Set  2, 
consisting  of  8  gambles,  was  always  responded  to  second.  Since  the 
for-sure  options  in  the  first  gambles  contained  all  of  the  CPAs  between 
0.5  and  3.5  (at  0.5  Intervals),  a  utility  function  could  be  computed 
from  the  indifference  probabilities  for  Set  1.  However,  this  was 
not  possible  for  the  Set  2  gabbles.  Thus,  for  some  of  the  subjects 
the  fitted  utility  function  was  based  on  performance  under  Format  1 
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while  for  the  remainder  of  the  subjects  the  function  was  based  on 
performance  under  Format  4.  Consequently,  there  appear  to  be  half 
as  many  subjects  per  format  in  Table  7  than  in  Table  S. 

The  40  observations  on  GADU  are  presented  in  Table  7  along  with 
the  twelve  cell  means  when  GADU  was  broken  down  by  Procedure,  Format, 
and  Sex.  Table  8  provides  aunmary  data.  The  observations  on  GADU 
were  essentially  normally  distributed  with  a  swan  of  .3384  and  a 
standard  deviation  of  .18.  They  ranged  from  .1153  (very  close 
approximation  of  stated  utility  function)  to  .8104  (very  poor 
approxisuition) . 

Main  Effects  for  GADU 

Procedure 

The  Procedure  effect  is  particularly  Interesting  because  of  the 
ordering  of  the  effectiveness  of  the  procedures  in  eliciting  a  small 
GADU.  As  Table  7  Indicates,  the  RC  procedure  produced  the  smallest 
GADU.  The  LC  and  SFS  procedures  were  clearly  less  effective  in 
obtaining  utilities  which  were  similar  to  the  atudenta'  stated  utility 
functions.  It  might  be  claimed  that  the  checking  devices  offered  by 
the  two  coherence  procedures  were  helpful  to  subjects  in  the  sense 
of  making  the  probabilities  upon  which  their  utilities  were  based 
conform  to  the  GPA  utilities  they  had  in  mind.  It  should  also  be  noted 
that  the  RC  procedure  led  to  better  results  on  both  the  MAE  and  the 
GADU  criteria  which,  as  mentioned  above,  are  hypothesised  to  measure 
two  distinct  skills. 

Format 

The  Format  effect  was  once  again  very  small,  as  demonstrated  by 
Tables  7 end  8.  Since  utilities  were  computed  only  from  the  first 


•even  |nblM,  only  on«  auaurt  of  GADU  was  obtained  for  Mch  subject, 
thereby  permitting  the  three-feetor  analysis  susssarlzed  In  the  two 
tablea.  The  aean  GADU  for  the  subjects  in  tha  Format  A  group  was  only 
slightly  lower  than  that  for  the  subjects  In  the  Forest  1  group 
(xf^-.3448  va.  Xyj«  .3597). 

Sex  of  Subject 

The  subject's  sex  once  again  had  an  iaportant  effect  on  the 
criterion  variable.  Perhaps,  the  aost  Interesting  observation  that 
say  be  sade  here  is  that  sales  (with  a  aean  GADU  of  .31S2)  were 
clearly  superior  to  feaeles  (with  a  Man  GADU  of  .3836)  in  minimizing 
the  difference  between  their  stated  and  fitted  utility  functions 
(See  Table  7).  This  contrasts  with  the  superior  feMle  performance 
on  obtaining  consistent  results  as  revealed  by  the  MAE  index. 

Interaction  Effects 

Procedura  by  Fonsst  Interaction 

The  complexity  of  this  interaction  for  dependent  variable  GADU 
was  somewhat  greater  than  for  MAE  (see  Table  8A).  For  the  RC  Procedure, 
perfoTMnce  did  not  differ  as  a  function  of  fonsst.  The  LC  procedure 
seeMd  better  in  conjunction  with  Fonsst  1  than  Fonwt  4,  while  the 
reverse  was  true  for  the  SFS  procedure. 


Procedure  by  Sex  Interaction 

A  smII  interaction  was  noted  between  Procedure  and  Sex,  as 
Indicated  in  Table  8C.  The  results  show  that  performance  under  the  LC 
procedure  evidenced  the  greatest  variation  as  a  function  of  sax.  Tha 
GADU  results  differ  from  the  MAE  results  in  that  GADU  shows  EC  to  be 
best  for  both  Mies  and  feMles  while  MAE  indicates  that  females 
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perfora  best  under  RC  and  males  par form  base  under  LC. 

Seat  bv  Format  Interaction 

This  interaction  was  vary  slight.  Faaalas  performed  only  somewhat 
better  under  Foraat  4  than  under  Format  1  while  there  was  essentially 
no  difference  in  male  performance  as  a  function  of  format.  GADU  and 
MAE  were  consistent  in  showing  that  performance  under  each  format  did 
not  differ  as  a  function  of  the  sex  of  the  subject. 

Procedure  by  Format  by  Sex  Interaction 

The  three-way  interaction  was  large  and  striking  as  Illustrated 
in  Figure  1.  The  effect  of  format  varied  as  a  function  of  sex  and 
utility  assessment  procedure. 

Effect  of  Statistical  Training  on  Performance 

It  was  hypothesised  that  students  who  had  had  some  statistical 
training  would  perform  better  than  those  who  had  had  none.  Indeed, 
the  14  students  who  had  had  one  or  more  hours  of  statistics  obtained 
an  average  MAE  of  .3411  while  those  with  zero  homes  of  statistics  obtained 
an  average  MAE  of  .4012.  When  GADU  was  the  criterion,  the  students  with 
statistical  training  obtained  an  average  score  of'  .3061  while  those  with  no 
training  obtained  an  average  score  of  .3583. 

General  Discussion 

Several  conclusions  may  be  drawn  from  the  three  experiments 
conducted.  First,  adjacent  and  distent  symmetric  gambles  appear  to 
be  more  useful  than  utility  gambles.  Second,  the  Regional  Coherence 
procedure  was  found  to  lead  to  superior  results  on  both  the  MAS  and 
GADU  criteria.  Finally,  neither  Foraat  provided  utilities  more  congruent 
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vith  initially  stated  utilities  than  the  other.  Mo  conclusion  could 
be  drawn  as  to  whether  the  fault  lay  in  the  initially  stated  or 
experimentally  elicited  utilities. 

When  taken  together,  however,  the  Regional  Coherence  procedure 
and  the  modified  ends-in  format  (Format  4)  seen  promising,  at  least 
for  people  with  moderate  to  no  statistical  training.  Unfortunately, 
while  the  undergraduates  participating  in  thla  experiment  came  from 
a  wide  range  of  academic  disciplines,  they  nay  have  lacked  either 
the  ability  or  motivation  to  fully  appreciate  and  understand  the 
complexity  and  importance  of  the  gambling  situations.  A  more 
statistically  inclined  sample  of  people  in  another  environment  might 
well  get  the  best  results  with  a  different  combination  of  procedure 
and  format. 

While  even  the  trained  statistician  could  no  doubt  benefit  from 
the  use  of  a  utility  assessment  procedure  with  coherence-checking 
capabilities,  the  "naive"  person  would  seem  to  gain  the  most  from 
use  of  such  a  procedure  in  the  interactive  environment  of  CADA.  And 
for  these  persons  Experiment  3  suggests  that  Regional  Coherence  with 
a  graphic  enda-in  method  of  eliciting  indifference  probabilities, 
if  not  the  ultimate  answer,  is  at  least  somewhat  more  profitable  than 
its  rivals.  We  feel  confident  that  this  combination  or  some  derivative, 
when  used  to  assess  utilities  in  a  real-life  situation,  will  prove 


useful . 


Appendix  A 

Evolution  of  the  Formats 
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Fomat  1  -  Direct  Elicitation 

1.  As  originally  on  CADA: 


FOR 

GAMBLES 

P  THAT  MAKES 

SURE 

P 

I-P 

YOU  INDIFFERENT 

0.50 

1.00 

0.00 

?  .5 

1.00 

1.50 

0.50 

?  — 

for  Pilot 

Experiment  1A: 

i 

CPA 

CPA 

I 

I 

I 

P 

1.0 

I 

I 

I  FOR  SURE  0.5 

GAMBLE  I 

I 

I 

1-P 

0.0 

I 

I 

What  value  of  P  makes  you  lndlfferaat  between  the 
for-sure  and  the  chance  options? _ 


3.  As  for  Pilot  Experiment  IB: 


Format 

One 

I  GPA  I 

I 

GPA  I 

I  I 

I 

3.5  P  I 

I  FOR  SURE  2.5  1 

I 

CHANCE 

I 

I  I 

I 

1.5  1-P  I 

What  value  of  p  makes  you 

indifferent  between  the  for- 

and  the  chance  options?  _ 

— 

for  Experiment  3: 

Format 

One 

FOR  SURE  OPTION 

CHANCE  OPTION 

I  CPA  X 

X  GPA 

I 

I  I 

I 

3.5 

PX  I 

I  2.5  I 

X 

X 

I  I 

1 

1.5 

A00X-P  I 

What  value  of  p  makes  you 

indifferent  between  the  for 

and  the  chance  options? _ 

MM 

Format  2  -  Ends- In 


1.  As  originally  on  CADA: 

OPTION: 

0  INDIFFERENT 

1  FOk  SURE 

2  CHANCE 

3  RESTART 


Which  would  you  prefer  if  p  •  .XX  ? 


2.  As  for  Pilot  Experiment  1A: 


I 

1 

I  FOR  SURE  0.S0 

I 

I 


1.00 


GAMBLE 


1-P  0.00 


Format  Two 


I  GPA  I 

I  I 

1  FOR  SURE  2.5  1 

I  I 


I 

CPA 

I 

OPTION 

0  INDIFFERENT 

I 

3.5 

P  I 

1  FOR  SURE 

I  CHANCE 

1 

2  CHANCE 

I 

1.5 

1-P  I 

3  RESTART 

Which  option  would  you  prefer  if  p  ■  .XX? 


Format  j  -  Quartering 


1.  Original  Conception: 


I 

GPA 

I 

I 

GPA 

I 

OPTION 

I 

I 

I 

3.5 

75Z 

I 

1  FOR  SURE 

I  FOR  SURE 

2.5 

I 

I 

CHANCE 

I 

2  CHANCE 

I 

I 

I 

1.5 

25Z 

I 

3  RESTART 

Which  option  do  you  prefer?. 


I 

GPA 

I 

I 

CPA 

I 

I 

I 

I 

3.5 

P 

I 

I  FOR  SURE 

2.5 

I 

I 

CHANCE 

I 

I 

I 

I 

1.5 

Q 

I 

Choose  the  set  of  P  and  Q  values  that  will  make  you  Indifferent 
between  the  for-sure  and  the  chance  options. 

P  75Z  70Z  65Z  60S  55Z  50Z 

Q  25%  30Z  35Z  40Z  A5Z  50Z 

The  P  value  that  makes  you  Indifferent  is? _ 
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2.  As  for  Pilot  Experiment  2: 


Format  Three 


FOR  SURE  OPTION 

A 

CHANCE  OPTION 

I 

CPA  1 

I 

CPA 

CHANCE 

X 

OPTION 

1 

X 

I 

3.5 

752 

I 

1 

TOR  SUR1 

1 

2.5  TOR  SURE  X 

X 

I 

2 

CHANCE 

1 

X 

X 

1.5 

252 

X 

3 

RESTART 

Which  option  do  you  prefer? _ 

- 

FOR  SURE  OPTION 

B 

CHANCE  OPTION 

I 

CPA  1 

1 

CPA 

CHANCE 

I 

I 

I 

I 

3.5 

P 

I 

I 

2.5  FOR  SURE  I 

I 

• 

I 

1 

X 

I 

1.5 

Q 

I 

Choose  the  set  of  P  and  Q  values  that  will  make  you  indifferent 
between  the  for-sure  and  the  chance  options. 

P  7 32  702  652  602  552  502 

Q  252  302  352  402  452  502 

The  P  value  that  makes  you  indifferent  is  7  _ 


Format  4  -  Modified  Ends- In 


1.  Original  Conception: 

Format  Four 


1002 

A  202 

802 

OPTION 

I 

I 

I 

I 

CPA 

2.5 

I 

I 

I 

I 

1GPA  I 

I  I 
11.5  X 

I  I 

CPA 

3.5 

I 

I 

I 

I 

0  INDIFFERENT 

1  FOR  SURE 

2  CHANCE 

3  RESTART 

FOR  SURE 

Which  option  do  you  prefer? 


CHANCE 


B 


100X 

100X-P 

P 

CPA 

I 

I  CPA 

CPA 

X 

I 

I 

I 

2.5 

X 

11.5 

3.5 

X 

I 

1 

X 

SURE  OPTION 

CHANCE  OPTION 

Your  indifference  probability,  P,  has  been  determined  to 
between  4  OX  and  60X. 

Value  of  P? 
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2.  As  for  Pilot  Experiment  2  and  for  Experiment  3: 


Format  Four 
A 


FOR  SURE  OPTION 

CHANCE  OPTION 

I 

CPA 

I 

I  CPA  I 

CPA 

I 

I 

I 

I  I 

I 

I 

2.5 

I 

11.5  I 

3.5 

I 

I 

I 

I  I 

I 

100Z 

20Z 

80Z 

OPTION 

0  INDIFFERENT 

1  FOR  SURE 

2  CHANCE 

3  RESTART 

Which  option  do  you  prefer? _ 


B 


FOR  SURE  OPTION 

OUNCE 

OPTION 

I 

CPA 

T 

A 

I CPA 

CPA 

I 

I 

I 

I 

I 

I 

2.5 

I 

11.5 

3.5 

I 

I 

I 

I 

I 

100Z 

100Z-P 

P 

Your  indifference  probability,  P,  has  been  determined  to  be 
between  40Z  and  60Z. 


Value  of  P? 


Appendix  B  -  Instructions 
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PILOT  EXPERIMENT  1 
A.  Introduction: 


experiment  1A 

IHE  PURPOSE  OF  THIS  EXPERIMENT  IS  TO  STUDY  GAMBLING  BEHAVIOR  WHEN  THE  OBJECT 
OF  THE  GAMBLE  IS  AN  IMPORTANT  FACTOR  RELATING  TO  A  COLLEGE  STUDENT'S  FUTURE 
OPPORTUNITIES ?  NAMELY ?  CUMULATIVE  GRADE  POINT  AVERAGE  <GPA>  AT  GRADUATION. 


YOU  WILL  BE  PRESENTED  WITH  SEVERAL  SITUATIONS  INVOLVING  CHOICES  BETWEEN  GAMBLES 
AND  SURE  THINGS.  YOU  ARE  TO  IMAGINE  THAT  THE  OUTCOME  OF  THE  OPTION  YOU  CHOOSE 
WILL  RESULT  IN  THE  FIXING  OF  A  PARTICULAR  CPA  ON  YOUR  PERMANENT  UNIVERSITY 
RECORD . 


WHILE  WE  REALIZE  THAT  YOU  ARE  NOT  ACCUSTOMED  TO  GAMBLING  ON  YOUR  GRADE  POINT 
AVERAGE t  WE  WOULD  L.IKL  YOU  TO  TRY  10  RESPOND  AS  YOU  WOULD  IF  THESE  WERE  ACTUAL 
SITUATIONS.  CONSIDER  CAREFULLY  WHAT  EACH  GRADE  POINT  AVERAGE  WOULD  MEAN  TO  YOU 
AT  GRADUATION  AND  HOW  EACH  WOULD  AN  EC l  YOUR  GOALS  FOR  FURTHER  EDUCATION  AND 
FUTURE  EMPLOYMENT. 


THESE  GAMBLES  CANNOT  ?  OF  COURSE r  BE  CARRIED  OUT  IN  REAL  LIFEf  HOWEVER?  THE 
SITUATION  SHOULD  BE  MEANINGFUL  TO  YOU.  THE  PURPOSE  OF  THE  EXPERIMENT  IS  TO 
STUDY  GAMBLING  BEHAVIOR  WHEN  THE  OUTCOMES  HAVE  SUBSTANTIAL  IMPORTANCE  RELAT¬ 
ING  TO  A  STUDENT'S  FUTURE. 


experiment  IB 


THE  PURPOSE  OF  THIS  EXPERIMENT  IS  TO  STUDY  GAMBLING  BEHAVIOR  WHEN  THE  OBJECT 
OF  THE  GAMBLE  IS  AN  IMPORTANT  FACTOR  RELATING  TO  THE  DUALITY  OF  THE  GRADUATE 
STUDENTS  IN  A  PHD  PROGRAM?  NAMELY*  THE  AVERAGE  OF  THEIR  VERBAL  AND  QUANTI¬ 
TATIVE  GRADUATE  RECORD  EXAMINATION  (GRE)  SCORES.  YOU  ARE  TO  THINK  OF  YOUR¬ 
SELF  AS  THE  CHAIR  OF  THE  ADMISSIONS  COMMITTEE  OF  THE  GRADUATE  DEPARTMENT  WITH 
WHICH  YOU  CAN  MOST  COMFORTABLY  IDENTIFY. 


AS  CHAIR  OF  THE  COMMITTEE?  YOUR  OPINIONS  ARE  HIGHLY  REGARDED  AND  THE  STUDENTS 
YOU  RECOMMEND  ARE  LIKELY  TO  BE  SENT  ACCEPTANCE  LETTERS.  YOU?  OF  COURSE? 

WOULD  LIKE  TO  ADMIT  THE  STUDENTS  WITH  THE  HIGHEST  AVERAGE  GRE  SCORES?  GIVEN 
THAT  ALL  OTHER  RATING  FACTORS  ARE  EQUAL. 


FOR  THE  PURPOSES  OF  THIS  EXPERIMENT?  HOWEVER?  A  MORE  DIFFICULT  TASK  WILL  BE 
PRESENTED  TO  YOU.  YOU  CANNOT  SIMPLY  RECOMMEND  THE  STUDENTS  WITH  THE  TOP 
TEST  SCORES.  RATHER?  YOU  WILL  BE  PRESENTED  WITH  SEVERAL  SITUATIONS  INVOLVING 
CHOICES  BETWEEN  GETTING  A  STUDENT  WITH  A  PARTICULAR  AVERAGE  GRE  SCORE  FOR 
SURE?  AND  A  GAMBLE  WHICH  WILL  GET  YOU  A  STUDENT  WITH  A  HIGHER  SCORE  IF  YOU  WIN 
OR  ONE  WITH  A  LOWER  SCORE  IF  YOU  LOSE. 
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WHILE  WE  REALIZE  THAT  YOU  ARE  NOT  ACCUSTOMED  TO  GAMBLING  ON  THE  GRE  SCORES 
OF  THE  STUDENTS  TO  BE  ADMITTED  TO  YOUR  GRADUATE  PROGRAM*  WE  WOULD  LIKE  YOU  TO 
RESPOND  AS  YOU  WOULD  IF  THESE  WERE  ACTUAL  DECISION  SITUATIONS.  CONSIDER 
CAREFULLY  WHAT  IT  WOULD  MEAN  TO  YOU*  YOUR  UNIVERSITY  *  AND  ALL  OTHERS  CONCERNED 
TO  ADMIT  A  GRADUATE  STUDENT  WITH  A  PARTICULAR  AVERAGE  GRE  SCORE. 


THE  SITUATIONS  THAT  WILL  BE  PRESENTED  TO  YOU  CANNOT*  OF  COURSE*  BE  CARRIED 
OUT  IN  REAL  LIFE*  HOWEVER*  YOUR  RESPONSES  TO  THESE  IDEALIZED  SITUATIONS  WILL 
TELL  US  SOMETHING  ABOUT  YOUR  BEHAVIOR  IN  MORE  REALISTIC  SITUATIONS.  THE 
PURPOSE  OF  THE  EXPERIMENT  IS  TO  STUDY  GAMBLING  BEHAVIOR  WHEN  THE  OUTCOMES  HAVE 
SUBSTANTIAL  IMPORTANCE  RELATING  TO  THE  NORMAL  FUNCTIONING  OF  A  GRADUATE 
INSTITUTION. 


B.  General  Instructions  -  Both  Formats 


IN  THIS  EXPERIMENT  YOU  WILL  BE  PRESENTED  WITH  SEVERAL  HYPOTHETICAL  CHOICE  SITU¬ 
ATIONS  CONSISTING  OF  A  FOR-SURE  AND  A  CHANCE  OPTION.  THE  FOR-SURE  OPTION 
OFFERS  YOU  THE  CERTAINTY  OF  KNOWING  THE  OUTCOME  IF  YOU  CHOOSE  THAT  OPTION. 

THE  CHANCE  OPTION  OFFERS  YOU  A  PROBABILISTIC  CHANCE  AT  TWO  POSSIBLE  OUTCOMES. 


FOR  EXAMPLE*  THE  FOR-SURE  OPTION  COULD  HAVE  YOU  ADMIT  A  STUDENT  WITH  AN 
AVERAGE  GRE  SCORE  OF  550.  THE  CHANCE  OPTION  COULD  THEN  OFFER  YOU  THE  CHANCE 
OF  ADMITTING  SOMEONE  WITH  A  SCORE  OF  750  WITH  PROBABILITY  P  (WHERE  P  CAN  BE 
ANY  VALUE  BETWEEN  ZERO  AND  ONE)  OR  A  PERSON  WITH  A  SCORE  OF  500  WITH  PROBA¬ 
BILITY  1-P. 


ASSUMING  THAT  YOU  PREFER  TO  ADMIT  A  STUDENT  WITH  A  SCORE  OF  750  TO  ONE  WITH  A 
SCORE  OF  550  TO  ONE  WITH  A  SCORE  OF  500  AND  ASSUMING  THAT  YOU  ARE  GIVEN  A 
CHOICE  BETWEEN  THE  FOR-SURE  AND  CHANCE  OPTIONS*  YOUR  DECISION  AS  TO  WHICH 
OPTION  YOU  WOULD  PREFER  TO  TAKE  WILL  DEPEND  UPON  THE  VALUE  OF  P*  THE  PROBA¬ 
BILITY  OF  ADMITTING  A  STUDENT  WITH  AN  AVERAGE  GRE  SCORE  OF  750  UNDER  THE 
CHANCE  OPTION. 


THIS  PARTICULAR  CHOICE  OF  ADMITTING  A  STUDENT  WITH  A  SCORE  OF  550  FOR  SURE 
VERSUS  THE  CHANCE  OF  ADMITTING  STUDENTS  WITH  SCORES  OF  750  AND  500  WITH 
PROBABILITIES  P  AND  1-P*  RESPECTIVELY*  WOULD  BE  DISPLAYED  ON  THE  SCREEN 
AS  FOLLOWS. 


GRE 

I 

I 

GRE 

I 

I 

I 

750 

P 

I 

FOR  SURE 

550 

I 

I  CHANCE 

I 

I 

I 

500 

1-P 

I 

I 

I 

I 
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FOR  P  VALUES  CLOSE  TO  ONE  YOU  WILL  PREFER  THE  CHANCE  OPTION  BECAUSE  IT  HILL 
PROBABLY  RESULT  IN  THE  ADMITTANCE  OF  A  STUDENT  WITH  AN  AVERAGE  ORE  SCORE  OF 
7S0  RATHER  THAN  THE  ADMITTANCE  OF  ONE  WITH  A  SCORE  OF  500.  THE  FOR-SURE 
OPTION.  ON  THE  OTHER  HAND.  SAYS  THAT  THE  ADMITTED  STUDENT  WILL  ONLY  HAVE  A 
SCORE  OF  550. 

FOR  VALUES  NEAR  ZERO.  HOWEVER.  YOU  WILL  PREFER  THE  SURE  THING  (ADMITTING 
A  STUDENT  WITH  A  SCORE  OF  550)  BECAUSE  THE  CHANCE  OPTION  WILL  PROBABLY  RESULT 
IN  THE  ADMITTANCE  OF  A  STUDENT  WITH  A  ORE  SCORE  OF  500.  THIS  IS  BECAUSE  A  P 
VALUE  NEAR  ZERO  MEANS  THAT  THE  VALUE  OF  1-P.  THE  PROBABILITY  OF  ADMITTING 
SOMEONE  WITH  AN  AVERAGE  GRE  SCORE  OF  500.  WILL  BE  NEAR  ONE. 

FOR  SOME  UNIQUE  INTERMEDIATE  VALUE  OF  P  YOU  SHOULD  BE  INDIFFERENT  BETWEEN 
THE  CHANCE  AND  THE  FOR-SURE  OPTIONS.  THIS  MEANS  THAT  WHEN  P  IS  AT  THIS  VALUE 
YOU  WOULD  BE  CONTENT  TO  HAVE  THE  OPTION.  EITHER  FOR-SURE  OR  CHANCE.  YOU 
RECEIVE  DETERMINED  BY  CHANCE. 

WE  ARE  INTERESTED  IN  THIS  VALUE  OF  P.  THE  INDIFFERENCE  PROBABILITY.  FOR  EACH 
SITUATION  (SET  OF  CHANCE  AND  FOR-SURE  OPTIONS)  WITH  WHICH  YOU  WILL  BE  PRE¬ 
SENTED. 


ALL  OF  THE  SITUATIONS  WITH  WHICH  YOU  WILL  BE  PRESENTED  WILL  INVOLVE  AVERAGE 
GRE  SCORES  AND  THESE  SCORES  ARE  MEANT  TO  REFER  TO  THE  TEST  SCORES  OF  PRO¬ 
SPECTIVE  GRADUATE  STUDENTS  IN  THE  DEPARTMENT  FOR  WHICH.  IN  THIS  EXPERIMENT. 
YOU  ARE  SERVING  AS  CHAIR  OF  THE  ADMISSIONS  COMMITTEE.  PLEASE  THINK  CAREFULLY 
HOW  THE  ADMITTANCE  OF  A  STUDENT  WITH  A  PARTICULAR  AVERAGE  GRE  SCORE  WILL 
AFFECT  YOURSELF.  YOUR  DEPARTMENT.  AND  ALL  OTHERS  CONCERNED. 


C.  Specialized  Instructions  -  To rest  One 


IN  THIS  HALF  OF  THE  EXPERIMENT  THE  COMPUTER  WILL  GENERATE  THE  SITUATIONS  AND 
YOU  ARE  TO  TELL  THE  EXPERIMENTER  THE  VALUE  OF  P  THAT  WILL  MAKE  YOU  INDIF¬ 
FERENT  BETWEEN  THE  FOR-SURE  AND  CHANCE  OPTIONS.  REMEMBER.  WHEN  THE  PROBA¬ 
BILITY  P  OF  ADMITTING  THE  STUDENT  WITH  THE  HIGHER  AVERAGE  GRE  SCORE  IN  THE 
CHANCE  OPTION  IS  AT  YOUR  INDIFFERENCE  PROBABILITY.  YOU  WILL  LIKE  THE  CHANCE 
AND  THE  FOR-SURE  OPTIONS  EQUALLY  WELL. 


WE  REALIZE  THAT  YOUR  INDIFFERENCE  PROBABILITIES  WILL  PROBABLY  BE  SOMEWHAT 
VAGUE.  SINCE  THERE  ARE  NO  RIGHT  OR  WRONG  ANSWERS.  JUST  TRY  TO  RESPOND  AS 
DELIBERATELY.  CAREFULLY  AND  ACCURATELY  AS  YOU  CAN.  HOWEVER.  IF  YOU  MAKE  A 
MISTAKE  OR  CHANGE  YOUR  MIND.  DON'T  WORRY.  YOU  WILL  HAVE  AN  OPPORTUNITY  TO 
CHANGE  YOUR  RESPONSES  LATER. 
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Speclallzed  Instructions  -  Forms t  Two 


IN  THIS  HALF  OF  THE  EXPERIMENT »  THE  COMPUTER  MILL  GENERATE  A  POSSIBLE  VALUE 
FOR  P  AND  YOU  ARE  TO  INDICATE  TO  THE  RESEARCH  ASSISTANT  MHETHER  THIS  PROBABIL¬ 
ITY  MILL  MAKE  YOU  INDIFFERENT  BETMEEN  THE  FOR-SURE  AND  CHANCE  OPTIONS.  REMEM¬ 
BER  »  IF  YOU  ARE  INDIFFERENT  BETMEEN  THE  TMO  OPTIONS  YOU  MOULD  NOT  PREFER  TO 
TAKE  ONE  OPTION  OVER  THE  OTHER. 


IF  THE  P  VALUE  PRESENTED  DOES  NOT  MAKE  YOU  INDIFFERENT  BETMEEN  THE  OPTIONS.  YOU 
ARE  TO  INDICATE  MHICH  OPTION.  EITHER  FOR-SURE  OR  CHANCE.  YOU  MOULD  PREFER  TO 
TAKE.  THE  COMPUTER  MILL  CONTINUE  TO  GENERATE  P  VALUES  UNTIL  YOU  INDICATE  THAT 
THE  P  VALUE  PRESENTED  MAKES  YOU  INDIFFERENT  BETMEEN  THE  FOR-SURE  AND  CHANCE 
OPTIONS. 


ALTHOUGH  YOUR  INDIFFERENCE  PROBABILITIES  MILL  UNDOUBTEDLY  BE  SOMEMHAT  VAGUE. 

TRY  TO  ANSWER  AS  ACCURATELY  AS  POSSIBLE.  THERE  ARE  NO  RIGHT  OR  MRONG  ANSWERS. 
IF  YOU  MAKE  A  MISTAKE  OR  CHANGE  YOUR  MIND  AS  TO  MHICH  OPTION  YOU  PREFER.  INFORM 
THE  EXPERIMENTER  AND  SHE  WILL  MAKE  THE  APPROPRIATE  CHANGE.  IF  YOU  CHANGE  YOUR 
MIND  AFTER  STATING  THAT  YOU  ARE  INDIFFERENT.  YOU  MAY  CHANGE  THE  INDIFFERENCE 
PROBABILITY  LATER.  AFTER  YOU  HAVE  FINISHED  ALL  OF  THE  SITUATIONS. 


D.  Before  Seeing  the  Fitted  Probabilities 


SPECIFYING  INDIFFERENCE  PROBABILITIES  IS  A  DIFFICULT  TASK.  AND  MOST  PEOPLE. 
INCLUDING  TRAINED  STATISTICIANS.  ARE  UNABLE  TO  PROVIDE  A  SET  OF  PROBABILITIES 
THAT  ARE  CONSISTENT  MITH  EACH  OTHER  ON  THE  FIRST  TRY.  USING  PROBABILITIES 
THAT  ARE  INCONSISTENT  WILL  USUALLY  RESULT  IN  THE  ADMITTANCE  OF  STUDENTS  MITH 
A  LOWER  AVERAGE  GRE  SCORE  THAN  WOULD  BE  ADMITTED  IF  THE  PROBABILITIES  MERE 
CONSISTENT. 


SINCE  THE  AVERAGE  GRE  SCORES  OF  THE  STUDENTS  YOU  ADMIT  HILL  HAVE  SUBSTANTIAL 
EFFECT  ON  THE  REPUTATION  OF  YOUR  DEPARTMENT.  IT  IS  VERY  IMPORTANT  TO  MAKE 
SURE  THAT  THE  INDIFFERENCE  PROBABILITIES  YOU  SPECIFY  ARE  CONSISTENT.  WITH  A 
LITTLE  HELP  THIS  IS  POSSIBLE. 


IN  ORDER  TO  HELP  YOU.  THE  COMPUTER  HAS  BEEN  PROGRAMMED  TO  PROVIDE  A  SET  OF 
INTERNALLY  CONSISTENT  PROBABILITIES  THAT  IS  SIMILAR  TO  THE  INCONSISTENT  SET 
YOU  SPECIFIED.  SUCH  AN  INTERNALLY  CONSISTENT  SET  OF  PROBABILITIES  IS  SAID  TO 
BE  COHERENT. 


THE  COMPUTER  HAS  FINISHED  CALCULATING  THE  COHERENT  SET  OF  INDIFFERENCE  PROB¬ 
ABILITIES.  EXAMINE  THEM  CAREFULLY  AND  COMPARE  THEM  TO  THE  ONES  YOU  SPECIFIED. 
IF  YOU  FEEL  THAT  THE  MEN  SET  OF  PROBABILITIES  ACCURATELY  REFLECTS  YOUR 
BETTING  BEHAVIOR  YOU  MAY  ACCEPT  THOSE  PROBABILITIES. 


SINCE  THE  COMPUTER  BASES  ITS  RECOMMENDED  PROBABILITIES  SOLELY  UPON  THE  INDIF¬ 
FERENCE  PROBABILITIES  SPECIFIED  AND  HAS  NO  REAL  IDEA  OF  THE  TRUE  FEELINGS 
ABOUT  ORE  SCORES  UNDERLYING  THEM*  THE  PROBABILITIES  MAY  NOT  ACCURATELY  REFLECT 
THOSE  FEELINGS. 


IF  YOU  FEEL  THAT  THIS  IS  THE  CASE  YOU  MAY  CHANGE  ANY  OR  ALL  OF  YOUR  SPECIFIED 
PROBABILITIES.  THE  COMPUTER  HILL  THEN  GENERATE  A  NEW  SET  OF  COHERENT  PROB¬ 
ABILITIES  BASED  ON  YOUR  REVISED  ESTIMATES. 


THE  FOLLOWING  TABLE  SUMMARIZES  THE  GAMBLES*  ALONG  WITH  THE  SPECIFIED  AND  COM¬ 
PUTER-GENERATED  INDIFFERENCE  PROBABILITIES.  COMPARE  THE  TWO  SETS  OF  PROB¬ 
ABILITIES  AND  DECIDE  WHETHER  OR  NOT  THE  COMPUTER-GENERATED  SET  ACCURATELY 
DESCRIBES  YUR  ATTITDES  TOWARD  THE  VARIOUS  TEST  SCORES. 


IF  SO*  YOU  MAY  DECIDE  TO  ACCEPT  THE  NEW  SET  OF  PROBABILITIES.  IF  NOT*  YOU  MAY 
CHANGE  SOME  OR  ALL  OF  THE  ORIGINAL  INDIFFERENCE  PROBABILITIES  YOU  SPECIFIED. 
THINK  CAREFULLY  BEFORE  YOU  MAKE  YOUR  DECISION. 
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PILOT  g|MgT  2 

The  instruction*  for  this  experiment  ware  bee  leal  ly  the  t— e  as 
those  for  Bxperlments  1  sad  3.  The  instructions  for  using  Format  Four 
(modified  eads-ln)  nay  bo  found  on  page  44.  The  Format  Throe 
(quartering  method)  explanation  is  reprinted  below. 


IN  THIS  HALF  OF  THE  EXPERIMENT  YOU  HILL  »E  PRESENTED  WITH  THE  FOLLOWING  KIND  OF 
TABLE*  REPRESENTING  ONE  SITUATION*  AND  YOU  ARE  TO  INDICATE  WHICH  OPTION*  EITHER 
FOR-SURE  OR  CHANCE*  YOU  WOULD  PREFER  TO  TAKE: 


FOR  SURE  OPTION  CHANCE  OPTION 


I 

GPA 

I 

I 

GPA 

CHANCE 

I 

I 

I 

I 

2 .  S 

8SX 

I 

I 

O.S  FOR  SURE 

I 

I 

I 

I 

I 

I 

0.0 

15Z 

I 

ALTHOUGH  YOUR  RESPONSES  WILL  PROBABLY  BE  SOMEWHAT  VAGUE  *  TRY  TO  ANSWER  AS 
ACCURATELY  AS  YOU  CAN.  THERE  ARE  NO  RIGHT  OR  WRONG  ANSWERS.  IF  YOU  MAKE  A 
MISTAKE  OR  CHANGE  YOUR  MIND  AS  TO  WHICH  OPTION  YOU  PREFER*  INFORM  THE  RESEARCH 
ASSISTANT  AND  SHE  WILL  MAKE  THE  APPROPRIATE  CHANGE. 
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ingggT  3 


A.  Introduction  -  SPS,  RC.  LC 

fHE  PURPOSE  OF  THIS  EXPERIMENT  IS  TO  STUDY  GAMBLING  BEHAVIOR  WHEN  THE  OBJECT 
)F  THE  GAMBLE  IS  AN  IMPORTANT  FACTOR  RELATING  TO  A  COLLEGE  STUDENT'S  FUTURE 
IPPORTUNITIES*  NAMELY »  CUMULATIVE  GRADE  POINT  AVERAGE  (GPA)  AT  GRADUATION* 


r0U  HILL  BE  PRESENTED  WITH  SEVERAL  SITUATIONS  INVOLVING  CHOICES  BETWEEN  GAMBLES 
(ND  SURE  THINGS*  YOU  ARE  TO  IMAGINE  THAT  THE  OUTCOME  OF  THE  OPTION  YOU  CHOOSE 
WILL  RESULT  IN  THE  FIXING  OF  A  PARTICULAR  GPA  ON  YOUR  PERMANENT  UNIVERSITY 
RECORD. 


WHILE  WE  REALIZE  THAT  YOU  ARE  NOT  ACCUSTOMED  TO  GAMBLING  ON  YOUR  GRADE  POINT 
AVERAGE*  WE  WOULD  LIKE  YOU  TO  TRY  TO  RESPOND  AS  YOU  WOULD  IF  THESE  WERE  ACTUAL 
ilTUATIONS.  CONSIDER  CAREFULLY  WHAT  EACH  GRADE  POINT  AVERAGE  WOULD  MEAN  TO  YOU 
Vr  GRADUATION  AND  HOW  EACH  WOULD  AFFECT  YOUR  GOALS  FOR  FURTHER  EDUCATION  AND 
FUTURE  EMPLOYMENT. 


t HESE  GAMBLES  CANNOT »  OF  COURSE*  BE  CARRIED  OUT  IN  REAL  LIFE!  HOWEVER*  THE 
SITUATIONS  SHOULD  BE  MEANINGFUL  TO  YOU.  THE  PURPOSE  OF  THE  EXPERIMENT  IS  TO 
;r»jDY  GAMBLING  BEHAVIOR  UHEN  THE  OUTCOMES  HAVE  SUBSTANTIAL  IMPORTANCE  RELAT¬ 
ING  TO  A  STUDENT'S  FUTURE. 


B.  General  Instructions  -  SPS,  LC 


IN  THIS  EXPERINENT  YOU  WILL  BE  PRESENTED  WITH  SEVERAL  HYPOTHETICAL  CHOICE  SITU¬ 
ATIONS  CONSISTING  OF  A  FOR-SURE  AND  A  CHANCE  OPTION.  THE  FOR-SURE  OPTION 
OFFERS  YOU  THE  CERTAINTY  OF  KNOWING  THE  OUTCOME  IF  YOU  CHOOSE  THAT  OPTION. 

THE  CHANCE  OPTION  OFFERS  YOU  A  PROBABILISTIC  CHANCE  AT  TWO  POSSIBLE  OUTCOMES. 


OR  EXAMPLE*  THE  FOR-SURE  OPTION  COULD  OFFER  YOU  A  OPA  OF  0.5.  THE  CHANCE 
OPTION  COULD  THEN  OFFER  YOU  A  CHANCE  OF  GETTING  A  OPA  OF  2.S  WITH  PROBABILITY 
P  (WHERE  P  CAN  BE  ANY  VALUE  BETWEEN  5X  AND  ?5X>  AND  A  CHANCE  ON  GETTING  A 
GPA  OF  0.0  WITH  PROBABILITY  100X-P. 


ASSUMING  THAT  YOUR  PREFERENCE  FOR  GPA'S  IS  2.5  OVER  0.5  OVER  0.0  AND 
4SSUMING  THAT  YOU  ARE  GIVEN  A  CHOICE  BETWEEN  THE  FOR-SURE  AND  THE  CHANCE 
OPTIONS*  YOUR  DECISION  AS  TO  WHICH  OPTION  YOU  WOULD  PREFER  TO  TAKE  WILL  DEPEND 
■PON  THE  VALUE  OF  P*  THE  PROBABILITY  OF  GETTING  A  OPA  OF  2.5  UNDER  THE 
< NANCE  OPTION. 


sit 
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•  ME  THANCE  and  for-sure  options  will  be  presented  to  you  in  separate  boxes  on 

THE  SCREEN  AND  THE  PROBABILITIES  ASSOCIATED  WITH  EACH  OF  THE  POSSIBLE  OPAf  IN 
•HE  CHANCE  OPTION  HILL  BE  CLEARLY  NARKED* 


A  LITTLE  THOUGHT  SHOULD  CONVINCE  YOU  THAT  THE  STATEHENTS  IN  THE  FOLLOWING 
THREE  PARAGRAPHS  ARE  TRUE.  HOWEVER.  THE  CONCEPTS  NAY  SEEN  SOHEWHAT  DIFFICULT 
SO  IF  YOU  HAVE  ANY  OUESTIONS  OR  ARE  AT  ALL  UNSURE  OF  WHAT  IS  BEING  STATED. 

please  bo  no  hesitate  to  ask  the  reseach  assistant  to  explain  things  none 

CLEARLY.  SHE  WILL  BE  GLAD  TO  CLARIFY  ANYTHING  YOU  DO  NOT  QUITE  UNDERSTAND. 


rt'R  v  VALUES  CLOSE  TO  100Z  YOU  WILL  PREFER  THE  CHANCE  OPTION  BECAUSE  IT  WILL 
PROBABLY  YIELD  YOU  A  GRADE  POINT  AVERAGE  OF  2.5  RATHER  THAN  A  GPA  OF  O.O. 
•HE  FOR-SURE  OPTION.  ON  THE  OTHER  HAND.  WILL  ONLY  NET  YOU  A  GPA  OF  0.5. 


FOR  VALUES  NEAR  OZ.  HOWEVER.  YOU  WILL  PREFER  THE  SURE  THING  (A  GPA  OF  0.5) 
CAUSE  THE  CHANCE  OPTION  WILL  PROBABLY  YIELD  YOU  A  GRADE  POINT  OF  0.0. 

HIS  IS  BECAUSE  A  p  VALUE  CLOSE  TO  0%  MEANS  THAT  THE  VALUE  OF  100Z-P.  THE 
^USABILITY  OF  UETTING  A  GPA  OF  0.0.  WILL  BE  NEAR  100Z 


•  UR  SOHE  UNIQUE  INTERMEDIATE  VALUE  OF  P  YOU  SHOULD  BE  INDIFFERENT  BETWEEN 
'HE  CHANCE  AND  THE  FOR-SURE  OPTIONS.  THIS  NEANS  THAT  WHEN  P  IS  AT  THIS  VALUE 
iOU  WOULD  BE  CONTENT  TO  LET  SOMEONE  ELSE  DETERMINE  WHICH  OPTION.  EITHER 
.  OR-SURE  OR  CHANCE.  YOU  RECEIVE. 


UL  ARE  INTERESTED  IN  THIS  VALUE  OF  P.  THE  INDIFFERENCE  PROBABILITY.  FOR  EACH 
SITUATION  (SET  OF  CHANCE  AND  FOR-SURE  OPTIONS)  WITH  WHICH  YOU  WILL  BE  PRE¬ 
SENTED. 


ML  OF  THE  SITUATIONS  WITH  WHICH  YOU  WILL  BE  PRESENTED  WILL  INVOLVE  GPA'S.  AND 
»«TE  FOLLOWING  SCALE  FOR  GRADE  POINTS  WILL  BE  USED: 


4.0*A 
3.0*B 
2.0-C 
1 .0*D 
0.0*F 


THESE  GPA'S  ARE  MEANT  TO  REFER  TO  YOUR  GPA  UPON  GRADUATION  FROM  YOUR  PRESENT 
■NIVERSITY.  IN  RESPONDING  TO  THE  CHOICES  THAT  WILL  BE  PRESENTED  TO  YOU. 
PLEASE  THINK  CAREFULLY  ABOUT  HOW  THE  ENTRY  OF  ANY  PARTICULAR  GPA  ON  YOUR  PER¬ 
MANENT  RECORD  WILL  AFFECT  YOUR  CAREER  AND  LIFE. 


General  Instructions  -  RC 


IN  THIS  EXPERIMENT  YOU  WILL  BE  PRESENTED  WITH  SEVERAL  HYPOTHETICAL  CHOICE  SITU 
AT  IONS  CONSISTING  OF  A  FOR-SURE  AND  A  CHANCE  OPTION.  THE  FOR-SURE  OPTION 
OFFERS  YOU  THE  CERTAINTY  OF  KNOWING  THE  OUTCOME  IF  YOU  CHOOSE  THAT  OPTION. 

THE  CHANCE  OPTION  OFFERS  YOU  A  PROBABILISTIC  CHANCE  AT  TWO  POSSIBLE  OUTCOMES. 


FOR  EXAMPLE*  THE  FOR-SURE  OPTION  COULD  OFFER  YOU  A  GPA  OF  2.5.  THE  CHANCE 
OPTION  COULD  THEN  OFFER  YOU  A  CHANCE  OF  GETTING  A  GPA  OF  3.5  WITH  PROBABILITY 
P  (WHERE  P  CAN  BE  ANY  VALUE  BETWEEN  5%  AND  95X)  AND  A  CHANCE  OF  GETTING  A 
GPA  OF  1.5  WITH  PROBABILTY  100X-P. 


ASSUMING  THAT  YOUR  PREFERENCE  FOR  UFA'S  IS  3.5  OVER  2.5  OVER  1.5  AND 
ASSUMING  THAT  YOU  ARE  GIVEN  A  CHOICE  BETWEEN  THE  FOR-SURE  AND  THE  CHANCE 
OPTIONS*  YOUR  DECISION  AS  TO  WHICH  OPTION  YOU  WOULD  PREFER  TO  TAKE  WILL  DEPEND 
UPON  THE  VALUE  OF  P*  THE  PROBABILITY  OF  GETTING  A  GPA  OF  3.5  UNDER  THE 
CHANCE  OPTION. 

HIE  CHANCE  AND  FOR-SURE  OPTIONS  WILL  BE  PRESENTED  TO  YOU  IN  BOXES  ON  THE 
SCREEN*  AND  THE  PROBABILITIES  ASSOCIATED  WITH  EACH  OF  THE  POSSIBLE  GPA'S  IN 
IHE  CHANCE  OPTION  WILL  BE  CLEARLY  MARKED. 


A  LITTLE  THOUGHT  SHOULD  CONVINCE  YOU  THAT  THE  STATEMENTS  IN  THE  FOLLOWING 
VHREE  PARAGRAPHS  ARE  TRUE.  HOWEVER*  IF  YOU  HAVE  ANY  QUESTIONS  OR  ARE  AT  ALL 
UNSURE  OF  WHAT  IS  BEING  STATED*  PLEASE  DO  NOT  HESITATE  TO  ASK  THE  RESEACH 
ASSISTANT  TO  EXPLAIN  THINGS  MORE  CLEARLY.  SHE  WILL  BE  GLAD  TO  CLARIFY  ANY¬ 
THING  THAT  YOU  DO  NOT  QUITE  UNDERSTAND. 

FOR  P  VALUES  CLOSE  TO  100%  YOU  WILL  PREFER  THE  CHANCE  OPTION  BECAUSE  IT  WILL 
PROBABLY  YIELD  YOU  A  GRADE  POINT  AVERAGE  OF  3.5  RATHER  THAN  A  GPA  OF  1.5. 
THE  FOR-SURE  OPTION*  ON  THE  OTHER  HAND*  WILL  ONLY  NET  YOU  A  GPA  OF  2.5. 


FOR  VALUES  NEAR  OX*  HOWEVER*  YOU  WILL  PREFER  THE  SURE  THING  (A  GPA  OF  2.5) 
BECAUSE  THE  CHANCE  OPTION  WILL  PROBABLY  YIELD  YOU  A  GRADE  POINT  OF  1.5. 

THIS  IS  BECAUSE  A  P  VALUE  CLOSE  TO  OX  MEANS  THAT  THE  VALUE  OF  100X-P*  THE 
PROBABILITY  OF  GETTING  A  GPA  OF  1.5*  WILL  BE  NEAR  100X. 


rOR  SOME  UNIQUE  INTERMEDIATE  VALUE  OF  P  YOU  SHOULD  BE  INDIFFERENT  BETWEEN 
T HE  CHANCE  AND  THE  FOR-SURE  OPTIONS.  THIS  MEANS  THAT  WHEN  P  IS  AT  THIS  VALUE 
YOU  WOULD  BE  CONTENT  TO  LET  SOMEONE  ELSE  DETERMINE  WHICH  OPTION*  EITHER 
) UR-SURE  OR  CHANCE*  YOU  RECEIVE. 


WE  ARE  INTERESTED  IN  THIS  VALUE  OF  P*  THE  INDIFFERENCE  PROBABILITY*  FOR  EACH 
SITUATION  (SET  OF  CHANCE  AND  FOR-SURE  OPTIONS)  WITH  WHICH  YOU  WILL  BE  PRE¬ 
SENTED. 


ALL  OF  THE  SITUATIONS  WITH  WHICH  YOU  WILL  BE  PRESENTED  WILL  INVOLVE  GPA'S*  AND 
THE  FOLLOWING  SCALE  FOR  GRADE  POINTS  WILL  BE  USED: 


4.0«A 
3 . 0*B 
2.0*C 
1 .  0  =  D 
0 . 0=F 


THESE  GPA'S  ARE  MEANT  TO  REFER  TO  YOUR  GPA  UPON  GRADUATION  FROM  YOUR  PRESENT 
UNIVERSITY.  IN  RESPONDING  TO  THE  CHOICES  THAT  WILL  BE  PRESENTED  TO  YOU* 
PLEASE  THINK  CAREFULLY  ABOUT  HOW  THE  ENTRY  OF  ANY  PARTICULAR  GPA  ON  YOUR  PER¬ 
MANENT  RECORD  WILL  AFFECT  YOUR  CAREER  AND  LIFE. 


C.  Procedure  Specific  Instructions  -  RC 

TO  HELP  YOU  DETERMINE  WHETHER  THE  INDIFFERENCE  PROBABILITIES  YOU  HAVE  SPECIFIED  i  j 
ARE  CONSISTENT  WITH  EACH  OTHER*  WE  HAVE  DEVISED  THE  FOLLOWING  PROCEDURE: 

AFTER  YOU  HAVE  CHOSEN  TWO  INDIFFERENCE  PROBABILITIES*  THE  COMPUTER  WILL  PRESENT 
YOU  WITH  A  TABLE  WHICH  DISPLAYS  THESE  TWO  SITUATIONS  AND  ALSO  TWO  MORE  SITU¬ 
ATIONS  FOR  WHICH  IT  HAS  DEDUCED  PROBABILTIES  BASED  ON  THE  TWO  YOU  SPECIFIED. 


THINK  AGAIN  OF  THE  SITUATION  IN  WHICH  YOU  WERE  OFFERED  A  'FOR-SURE'  GPA  OF  2.5 
AND  A  CHANCE  OF  GETTING  A  GPA  OF  3.5  WITH  PROBABILTY  P  AND  OF  GETTING  A  GPA  OF 
1.5  WITH  PROBABILITY  100Z-P.  LET  US  IMAGINE  THAT  YOU  SPECIFIED  P  TO  BE  60%. 

NOW  THINK  OF  A  SITUATION  IN  WHICH  YOU  ARE  OFFERED  A  'FOR-SURE'  GPA  OF  3.0  VERSUS 
A  CHANCE  OF  GETTING  A  GPA  OF  3.5  WITH  PROBABILITY  P  AND  OF  GETTING  A  GPA  OF 
>.5  WITH  PROBABILITY  100Z-P.  LET  US  IMAGINE  THAT  FOR  THIS  SITUATION  YOU  CHOOSE 
P  VALUE  OF  65%. 


SITUATIONS 

1 - 2 - 3 - 4 


P  CHANCE 

3.5 

3.5 

3.0 

3.5 

FOR  SURE 

2.5 

3.0 

2.5 

3.0 

t<)0%-P  CHANCE 

1.5 

P«60Z 

2.5 

P-65% 

1.5 

P-70Z 

1.5 

P-86X 

YOU  WILL  NOW  BE  INVITED  TO  EXAMINE  ALL  FOUR  SITUATIONS  AND  TO  DETERMINE 
WHETHER  THE  INDIFFERENCE  PROBABILITIES  DISPLAYED  FOR  EACH  ARE  CONSISTENT 
WITH  YOUR  VIEWS.  IF  YOU  RECOGNIZE  SOME  INCONSISTENCIES  YOU  WILL  BE  ABLE  TO 
ADJUST  THE  P  VALUES  FOR  ANY  TWO  OF  THE  SITUATIONS  (THEREBY  FIXING  THE  P  VALUES 
riiR  THE  OTHER  TWO  SITUATIONS).  YOU  WILL  BE  ASKED  TO  MANIPULATE  THE  P  VALUES 
UNTIL  YOU  ARE  REASONABLY  SATISFIED  WITH  THE  DISPLAYED  INDIFFERENCE  PROBABIL¬ 
ITIES  FOR  ALL  FOUR  SITUATIONS. 


IT  MAY  TAKE  SOME  TIME  TO  FIND  THE  P  VALUES  THAT  WILL  MAKE  YOU  SATISFIED  IN  ALL 
)F  THE  SITUATIONS*  SO  WE  HOPE  THAT  YOU  WILL  BE  PATIENT  AND  PERSEVERING. 
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WE  REALIZE  THAT  SPECIFYING  INDIFFERENCE  PROBABILITIES  FOR  ISOLATED  SITUATIONS 
IS  A  DIFFICULT  TASK  --  EVEN  TRAINED  STATISTICIANS  FIND  IT  DIFFICULT.  BECAUSE 
IT  IS  HARD  TO  SEE  HOW  THE  VARIOUS  SITUATIONS  RELATE  TO  EACH  OTHER*  THE  INDIF¬ 
FERENCE  PROBABILITIES  FOR  THEN  ARE  LIKELY  TO  BE  INCONSISTENT  WITH  EACH  OTHER. 
IF  YO'IR  FINAL  GPA  WERE  ACTUALLY  DETERMINED  BY  HAVING  YOU  PLAY  THROUGH  SEVERAL 
SITUATIONS*  IT  WOULD  BE  HIGHER  IF  THE  INDIFFERENCE  PROBABILITIES  FOR  THE 
VARIOUS  SITUATIONS  WERE  CONSISTENT  WITH  EACH  OTHER  THAN  IF  THEY  WERE  INCON¬ 
SISTENT.  A  SET  OF  C0NSISTEN1  PROBABILITIES  IS  SAID  TO  BE  COHERENT. 


SINCE  YOUR  FINAL  GPA  IS  LIKELY  TO  AFFECT  YOUR  FUTURE  OPPORTUNITIES*  IT  IS  IM¬ 
PORTANT  THAT  YOU  TRY  TO  MAKE  THE  INDIFFERENCE  PROBABILITIES  AS  CONSISTENT  WITH 
t ACH  OTHER  AS  YOU  CAN.  THE  PROCEDURE  EXPLAINED  ABOVE  WAS  DESIGNED  TO  HELP 
if OU  IN  THIS  TASK.  IF  YOU  ARE  INDIFFERENT  BETWEEN  THE  TWO  OPTIONS  PRESENTED 
-N  EACH  OF  THE  FOUR  SITUATIONS  FOR  THE  STATED  PROBABILITIES*  THEN  ALL  OF  THOSE 
INDIFFERENCE  PROBABILITIES  WILL  BE  CONSISTENT  WITH  EACH  OTHER. 


Procedure  Specific  Instructions  -  LC 

TO  HELP  YOU  DETERMINE  WHETHER  THE  INDIFFERENCE  PROBABILITY  YOU  HAVE  CHOSEN 
ACCURATELY  REFLECTS  YOUR  VIEWS.  WE  HAVE  PROVIDED  A  METHOD  BY  WHICH  YOU  CAN  TEST 
YOUR  INITIAL  JUDGMENT  AGAINST  A  JUDGMENT  IN  A  RELATED  SITUATION. 

THINK  AGAIN  ABOUT  THE  SITUATION  IN  WHICH  YOU  WERE  OFFERED  A  'FOR-SURE'  GPA  OF 
0.3  VERSUS  A  CHANCE  OF  GETTING  A  GPA  OF  2.5  WITH  PROBABILITY  P  AND  OF  GETTING  A 
GPA  OF  0.0  WITH  PROBABILITY  100X-P.  LET  US  IMAGINE  THAT  YOU  SPECIFIED  P  TO 
BE  SOX. 

THE  COMPUTER  PROGRAM  THEM  REPRODUCES  THIS  SITUATION  (WITH  P*50X>  AND  ALSO  PRE¬ 
SENTS  YOU  WITH  ANOTHER  TYPE  OF  CHOICE  SITUATION  CONSTRUCTED  OF  TWO  CHANCE 
OPTIONS*  WITH  THE  PROBABILITIES  FOR  THE  NEW  SITUATION  DEDUCED  FROM  THE  INDIF 
FERENCE  PROBABILITY  YOU  SPECIFIED  FOR  THE  FIRST  SITUATION. 


HERE  ARE  THE  TWO  SITUATIONS 

GAMBLE  SITUATION  1  GAMBLE  SITUATION  2 


I 

OPTION  1 

OPTION  2 

I 

I 

OPTION  1 

OPTION  2 

I 

(FOR  SURE) 

(CHANCE) 

I 

I 

I 

2.5  SOX 

I 

I 

50X 

2.5 

75X 

I 

0.5 

I 

I 

50% 

0.5 

— 

I 

0.0  50X 

I 

I 

—  — 

0.0 

25X 

RECALL  THAT  SITUATION  1  SAYS  THAT  YOU  HAVE  NO  PREFERENCE  BETWEEN  GETTING 
A  GPA  OF  O.S  FOR  SURE  AND  TAKING  A  CHANCE  OPTION  WHICH  GIVES  YOU  A  50X  CHANCE 
OF  GETTING  A  GPA  OF  2.5  AND  A  50X  CHANCE  OF  GETTING  A  GPA  OF  0.0. 

YOU  ARE  NOW  ASKED  TO  DETERMINE  WHETHER  OR  NOT  YOU  ARE  INDIFFERENT  BETWEEN  THE 
TWO  ORTIOMS  IN  SITUATION  2.  NOTE  THAT  OPTION  2  GIVES  YOU  A  BETTER  CHANCE  AT 
THE  MOST  PREFERRED  OUTCOME  (2.5)  BUT  ALSO  SOME  POSSIBILITY  OF  GETTING  THE 
LEAST  PREFERRED  OUTCOME  (0.0). 

THERE  IS  NO  CHANCE  OF  GETTING  THE  MIDDLE  GPA  (0.5).  OPTION  1*  ON  THE  OTHER 
HAND.  GIVES  YOU  A  CHANCE  OF  GETTING  THE  MIDDLE  GPA  INSTEAD  OF  A  CHANCE  OF  GET- 
•  IMG  THE  LOWEST  GPA.  THIS  IMPROVEMENT  (ASSUMING  YOU  PREFER  TO  HAVE  A  CHANCE 
Of  GETTING  THE  0.5  TO  A  CHANCE  OF  GETTING  THE  0.0) .HOWEVER*  IS  COMPENSATED 
H Y  THE  LOWER  PROBABILITY  OF  GETTING  THE  HOST  PREFERRED  GPA  (A  50X  CHANCE 
VERSUS  A  7SX  CHANCE). 
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YOU  WILL  BE  ASKED  TO  MANIPULATE  THE  CHANCE  OF  GETTING  A  GPA  OF  2.5  IN  SITUA¬ 
TION  It  OPTION  2  UNTIL  A)YOU  ARE  INDIFFERENT  BETWEEN  OPTIONS  1  AND  2  IN  SITUA¬ 
TION  2  AND  B>  YOU  ARE  ALSO  INDIFFERENT  BETWEEN  THE  FOR-SURE  AND  CHANCE  OPTIONS 
IN  SITUATION  1. 

IT  MAY  TAKE  SOME  TIME  BEFORE  YOU  ARE  SATISFIED  WITH  (I.E.*  INDIFFERENT  BETWEEN) 
BOTH  SITUATIONS.  SO  WE  HOPE  THAT  YOU  WILL  BE  PATIENT  AND  PERSEVERING. 


WE  REALIZE  THAT  SPECIFYING  INDIFFERENCE  PROBABILITIES  FOR  ISOLATED  SITUATIONS 
IS  A  DIFFICULT  TASK  —  EVEN  TRAINED  STATISTICIANS  FIND  IT  DIFFICULT.  BECAUSE 
IT  IS  HARD  TO  SEE  HOW  THE  VARIOUS  SITUATIONS  RELATE  TO  EACH  OTHERt  THE  INDIF¬ 
FERENCE  PROBABILITIES  FOR  THEM  ARE  LIKELY  TO  BE  INCONSISTENT  WITH  EACH  OTHER. 
IF  YOUR  FINAL  GPA  WERE  ACTUALLY  DETERMINED  BY  HAVING  YOU  PLAY  THROUGH  SEVERAL 
SITUATIONS.  IT  WOULD  BE  HIGHER  IF  INDIFFERENCE  PROBABILITIES  FOR  THE  VARIOUS 
SITUATIONS  WERE  CONSISTENT  WITH  E-ACH  OTHER  THAN  IF  THEY  WERE  INCONSISTENT.  A 
SEI  OF  CONSISTENT  PROBABILITIES  IS  SAID  TO  BE  COHERENT. 

SINCE  YOUR  FINAL  GPA  IS  LIKELY  TO  AFFECT  YOUR  FUTURE  OPPORTUNITIES.  IT  IS  IM¬ 
PORTANT  THAT  YOU  TRY  TO  MAKE  THE  INDIFFERENCE  PROBABILITIES  AS  CONSISTENT  WITH 
EACH  OTHER  AS  YOU  CAN.  THE  PROCEDURE  EXPLAINED  ABOVE  WAS  DESIGNED  TO  HELP  YOU 
IN  THIS  TASK.  IF  YOU  ARE  INDIFFERENT  BETWEEN  THE  OPTIONS  PRESENTED  IN  BOTH 
SITUATIONS  FOfLTHE  STATED  PROBABILITIES.  THEN  YOUR  INDIFFERENCE  PROBABILITIES 
WILL  BE  COHERENT. 


D.  Format  Instructions  -  Direct  Probability  Elicitations  (all  procedures) 


N  THIS  HALF  OF  THE  EXPERIMENT  THE  COMPUTER  WILL  GENERATE  THE  SITUATIONS  AND 
YOU  ARE  TO  TELL  THE  EXPERIMENTER  THE  VALUE  OF  P  THAT  WILL  MAKE  YOU  INDIF¬ 
FERENT  BETWEEN  THE  FOR-SURE  AMD  THE  CHANCE  OPTIONS. 


HERE  IS  AN  EXAMPLE) 

FOR  SURE  OPTION  CHANCE  OPTION 


I 

GPA 

I 

I 

GPA 

I 

I 

I 

2.5 

PX 

I 

0.5 

I 

I 

I 

I 

I 

© 

• 

o 

100%-P 

WHAT 

VALUE  OF  P  MAKES  YOU 

INDIFFERENT 

BETWEEN 

bi 

X 

t- 

FOR-SURE  AND 

THE  CHANCE  OPTIONS? 


REMEMBER »  WHEN  THE  PROBABILITY  P  OF  GETTING  THE  HIGHER  GPA  IN  THE  CHANCE  OPTION 
IS  AT  YOUR  INDIFFERENCE  PROBABILITY,  YOU  WILL  LIKE  THE  CHANCE  AND  THE  FOR-SURE 
OPTIONS  EQUALLY  WELL. 
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WE  REALIZE  THAT  YOUR  INDIFFERENCE  PROBABILITIES  WILL  UNDOUBTEDLY  BE  SOMEWHAT 
VAGUE.  SINCE  THERE  ARE  NO  RIGHT  OR  WRONG  ANSWERS ,  JUST  TRY  TO  RESPOND  AS 
DELIBERATELY.  CAREFULLY  AND  ACCURATELY  AS  YOU  CAN.  HOWEVER.  IF  YOU  MAKE  A 
MISTAKE  OR  CHANGE  YOUR  MIND.  DON'T  WORRY,  YOU  WILL  HAVE  AN  OPPORTUNITY  TO 
CHANGE  YOUR  RESPONSES  LATER. 


Format  Instructions  -  Modified  Ends-In  (all  procedures) 


IN  THIS  HALF  OF  THE  EXPERIMENT  YOU  WILL  BE  PRESENTED  WITH  THE  FOLLOWING  KIND 
OF  TABLE,  REPRESENTING  ONE  SITUATION,  AND  YOU  ARE  TO  INDICATE  WHICH  OPTION* 
EITHER  FOR-SURE  OR  CHANCE.  YOU  WOULD  PREFER  TO  TAKES 


FOR  SURE  OPTION  CHANCE  OPTION 


GPA 

1 

I 

GPA 

I 

GPA 

I 

I 

I 

I 

I 

0.5 

I 

I 

0.0 

I 

2.5 

I 

I 

I 

I 

I 

1002 

60* 

40* 

IF  YOU  HAVE  NO  PREFERENCE  BETWEEN  THE  OPTIONS*  STATE  THAT  YOU  ARE  INDIFFERENT. 
REMEMBER,  IF  YOU  ARE  INDIFFERENT  BETWEEN  THE  TWO  OPTIONS  YOU  WOULD  CONSENT  TO 
TO  LET  SOMEONE  ELSE  CHOOSE  WHICH  OPTION*  EITHER  FOR-SURE  OR  CHANCE*  YOU  RECEIVE. 
ALTHOUGH  YOUR  REPOMSCS  WILL  UNDOUBTEDLY  BE  RATHER  VAGUE*  TRY  TO  ANSWER  AS 
ACCURATELY  AS  YOU  CAN.  THERE  ARE  NO  RIGHT  OR  WRONG  ANSWERS.  IF  YOU  MAKE  A 
MISTAKE  OR  CHANGE  YOUR  HIND  AS  TO  WHICH  OPTION  YOU  PREFER*  INFORM  THE  RESEARCH 
ASSISTANT  AND  SHE  » ILL  MAKE  THE  APPROPRIATE  CHANGE.  YOUR  FIRST  SITUATION 
FOLLOWS. 


E.  Procedure  Specific  Instructions  -  SFS 


SPECIFYING  INDIFFERENCE  PROBABILITIES  IS  A  DIFFICULT  TASK*  AND  MOST  PEOPLE* 
INCLUDING  TRAINED  STATISTICIANS,  ARE  UNABLE  TO  PROVIDE  A  SET  OF  PROBABILITIES 
THAT  ARE  CONSISTENT  WITH  EACH  OTHER  ON  THE  FIRST  TRY.  USING  PROBABILITIES 
THAT  ARE  INCONSISTENT  WILL  USUALLY  RESULT  IN  THE  APPEARANCE  OF  A  LOWER  FINAL 
GPA  ON  YOUR  PERMANENT  RECORD  THAN  WOULD  APPEAR  IF  THE  PROBABILITIES  WERE 
CONSISTENT. 


:  INCE  YOUR  FINAL  GRADE  POINT  AVERAGE  WILL  GREATLY  AFFECT  YOUR  OPPORTUNITIES  FOR 
THE  FUTURE*  IT  IS  VERY  IMPORTANT  TO  MAKE  SURE  THAT  THE  INDIFFERENCE  PROB¬ 
ABILITIES  YOU  SPECIFY  ARE  CONSISTENT.  WITH  A  LITTLE  HELP  THIS  IS  POSSIBLE. 


IN  ORDER  TO  HELP  YOU,  THE  COMPUTER  HAS  BEEN  PROGRAMMED  TO  PROVIDE  A  SET  OF 
( NTERNALLY  CONSISTENT  PROBABILITIES  THAT  IS  SIMILAR  TO  THE  INCONSISTENT  SET 
YOU  SPECIFIED.  SUCH  AN  INTERNALLY  CONSISTENT  SET  OF  PROBABILITIES  IS  SAID  TO 
I<E  COHERENT. 
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GiT  BACK  AND  RELAX  FOR  A  FEW  MINUTES  WHILE  THE  COMPUTER  MAKES  THE  NECESSARY 
CALCULATIONS. 


rHE  COMPUTER  HAS  FINISHED  CALCULATING  THE  COHERENT  SET  OF  INDIFFERENCE  PROB¬ 
ABILITIES.  EXAMINE  THEM  CAREFULLY  AND  COMPARE  THEM  TO  THE  ONES  YOU  SPECIFIED. 
IF  YOU  FEEL  THAT  THE  NEW  SET  OF  PROBABILITIES  ACCURATELY  REFLECTS  YOUR 
BETTING  BEHAVIOR  YOU  MAY  ACCEPT  THOSE  PROBABILITIES. 


SINCE  THE  COMPUTER  BASES  ITS  RECOMMENDED  PROBABILITIES  SOLELY  UPON  THE  INDIF¬ 
FERENCE  PROBABILITIES  SPECIFIED  AND  HAS  NO  REAL  IDEA  OF  THE  TRUE  FEELINGS 
ABOUT  GRADE  POINT  AVERAGES  UNDERLYING  THEM*  THE  PROBABILITIES  MAY  NOT  ACCU¬ 
RATELY  REFLECT  THOSE  FEELINGS. 


IF  YOU  FEEL  THAT  THIS  IS  THE  CASE  YOU  MAY  CHANGE  ANY  OR  ALL  OF  YOUR  SPECIFIED 
PROBABILITIES.  THE  COMPUTER  WILL  THEN  GENERATE  A  HEW  SET  OF  COHERENT  PROB¬ 
ABILITIES  BASED  ON  YOUR  REVISED  ESTIMATES. 


THE  FOLLOWING  TABLE  SUMMARIZES  THE  GAMBLES.  ALONG  WITH  THE  SPECIFIED  AND  COM¬ 
PUTER-GENERATED  INDIFFERENCE  PROBABILITIES.  COMPARE  THE  TWO  SETS  OF  PROB¬ 
ABILITIES  AND  DECIDE  WHETHER  OR  NOT  THE  COMPUTER-GENERATED  SET  ACCURATELY 
DESCRIBES  YOUR  ATTITUDES  TOWARD  THE  VARIOUS  GRADE  POINT  AVERAGES. 


FF  SO.  YOU  MAY  DECIDE  TO  ACCEPT  THE  NEW  SET  OF  PROBABILITIES.  IF  NOT.  YOU  MAY 
:HANGE  SOME  OR  ALL  OF  THE  ORIGINAL  INDIFFERENCE  PROBABILITIES  YOU  SPECIFIED. 


HINK  CAREFULLY 

BEFORE  YOU  MAKE  YOUR 

DECISION 

♦ 

)T  TUATION 

FOR 

CHANCE 

INDIFF 

.  P 

NO. 

SURE 

P 

100%-P 

SPEC'D 

COMP 

1 

0.50 

1  .00 

0.00 

50% 

52% 

2 

1.00 

2.00 

0.00 

50% 

53% 

3 

1.50 

2.50 

0.50 

50% 

54% 

4 

2.00 

2.50 

1.50 

50% 

54% 

5 

2.50 

4.00 

1.00 

45% 

62% 

6 

3.00 

3.50 

2.50 

55% 

53% 

7 

3.50 

4.00 

3.00 

70% 

69% 

8 

1 . 00 

1.50 

0.50 

50% 

51% 

OPTIONS 

9 

1 .50 

2.00 

1.00 

50% 

52% 

1.  MODIFY  SPEC'D  P 

10 

1.50 

3.00 

0.00 

60% 

57% 

2.  ACCEPT  COMP  P'S 

n 

2.00 

3.00 

1  .00 

50% 

56% 

12 

2.00 

4.00 

0.00 

70% 

63% 

13 

2.50 

3.00 

2.00 

50% 

52% 

14 

2.50 

3.50 

1.50 

55% 

56% 

15 

3.00 

4.00 

2.00 

55% 

62% 

FPTION? 
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APPENDIX  C  -  Questionnaire 


1.  Hare  you  at  any  tine  unsure  of  what  was  expected  of  you? 
If  sot  when? 


2.  How  could  the  directions  have  been  changed  so  as  to  alleviate 
the  above  problem? 


Are  there  any  comments  you  would  like  to  make  about  the 
experiment  or  the  experimenter? 


(Experiment  3  only)  Please  give  some  indication  of  how  useful 
each  GPA  in  the  following  scale  is  to  you.  The  utility  (usefulness 
or  worth)  of  a  GPA  of  0.0  is  assumed  to  be  0  while  that  of  a  GPA 
of  4.0  is  100.  Please  specify  intermediate  values  for  the  other 
GPAs. 


utility  to  you 


4.0 


100 


Appendix  D 


1/4 

Format 

Order 

4/1 


Results  Tables  for  Experiment  3 


TABLE  4 


Overall  Mean  Absolute  Errors  for  Forty  Subjects  with  Breakdown 

by  Procedure  and  Sex 


MALES 

FEMALES 

RC 

SFS 

LC 

RC 

SFS 

i  “  . 

.6574 

.7736 

.5976 

.1311 

.2317 

.2972 

.3991 

.3883 

.1927 

.0555 

WEm 

.4372 

.6445 

.3433 

.4662 

.3237 

.6073 

.3383 

.8540 

.2099 

.5727 

.2037 

.3917 

.3550 

.4051 

.4936 

.2479 

.3516 

.3026 

.5020 

.3105 

.5067 

.2180 

.4409 

.3342 

.4059 

.4188 

.6506 

.6816 

X-.4803 

n-7 

X-.5046 

n-6 

x*. 3938 
n*6 

x-,2978 

n*6 

X-.3285 

n*8 

x~.4624 

n»7 

Grand  Mean  (M)  ■  .4065 
Procedure:  $ex: 


*RC 

.3923 

(n  -  13) 

Xyi  ■  .4554 

(n  -  19) 

*3FS  " 

x,  „  - 

.4021 

.4295 

(n  -  14) 

(n  -  13) 

Xy  -  .3623 

(n  -  21) 

-5i 


Forme 

1 


Forme 

4 


TABLE  7 

Greatest  Absolute  Differences  in  Utilities  (GADU)  for  Forty 
Subjects  with  Breakdowns  by  Procedure,  Forme  and  Sex 


|  HALES 

|  FEMALES  | 

RC 

LC 

SFS 

RC 

LC 

SFS 

.2129 

.1642 

.6096 

4039 

.1850 

.5449 

.1170 

.3699 

.3601 

1494 

.4958 

.5193 

.1757 

.2361 

.5596 

5620 

.4798 

.1872 

.2343 

x  -_.J685 

'w1?' 

V-'J098 

*n"-”3718 

V-'i869 

V.ln> 

.7205 

.2477 

.3635 

.2844 

.8104 

.3758 

.2763 

.3210 

.1178 

.2070 

.1759 

.4199 

.2145 

.5064 

.2234 

.1380 

.2826 

.6599 

.1153 

.2940 

.2165 

x-^316 

X-.3J84 

X-.2J49 

*nmm-  2p98 

V,3?08 

x  -  .4167 
n  ■  4 

Grand  Mean  (M)  -  .33*4 


Procedure: 

5^,  -  .2751  (n  -  13) 

\c  -  .3514  (n  -  13) 

x  -  .3848  (n  -  14) 
SFS 

Forme: 

xpi-  .3456  (n  -  19) 
.3319  (n  ■  21) 

Sex: 

-  .3111  (n  -  19) 
Xy  -  .  3631  (n  -  21) 
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TABLE  8 


Two-way  S usury  Tables  for  Dependent  Variable  -  GADU 


A.  S unwary  Table  for  Procedure  by  Format 


^  -  .3456 
n  -19 


XRC  ‘  2751  XLC 


‘  ‘35U  XSFS  "  *3848 


n  -13 


a  -13 


n  -  14 


ry  Table  for  Sex  by  Format 


MALES 


x  -  .3117  x  -  .3762 


n  -  9 


n  -  10 


xF1  -  .3456 
n  -  19 


x14  -  .3111 
n  -  19 


Xj  -  .3631 
n  -  21 


Summary  Table  for  Procedure  by  Sex 


V  3111 

a  -  19 
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1030  East  Green  Street 
Pasadena,  CA  91101 

1  Office  of  the  Chief  of  Naval  Operations 
Research  Development  A  Studies  Branch 
(OP-115) 

Washington,  DC  20350 

1  LT  Frank  C.  Petho,  MSC,  USN  (Ph.D) 

Code  L51 

Naval  Aerospace  Medical  Research  Laborat 
Pensacola,  FL  32508 

1  Mr.  Arnold  Rubensteln 

Naval  Personnel  Support  Technology 
Naval  Material  Command  (08T299) 

Room  1099,  Crystal  Plaza  #5 
2221  Jefferson  Davis  Highway 
Arlington,  VA  20360 


Navy 


1  Dr.  Robert  C.  Saith 

Office  of  Chief  of  Naval  Operations 
OP-987H 

Washington,  DC  20350 

1  Dr.  Alfred  F.  anode 

Training  Analyala  A  Evaluation  Group 
(TAEG) 

Dept,  of  the  Navy 
Orlando.  FL  32813 

1  Dr.  Richard  Sorensen 

Navy  Personnel  RAD  Center 
San  Diego.  CA  92152 

1  W.  Gary  Thomson 

Naval  Ocean  Systems  Center 
Code  7132 

San  Diego,  CA  92152 

1  Dr.  Ronald  We it an an 
COde  59  WZ 

Department  of  Administrative  Sciences 
U.  S.  Naval  Postgraduate  School 
Monterey,  CA  93990 

1  DR.  MARTIN  F.  WISKDFF 

NAVY  PERSONNEL  RA  D  CENTER 
SAN  DIEGO.  CA  92152 
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Technical  Director 

U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
50C1  Eisenhower  Avenue 
Alexandria,  VA  22333 

HQ  USAREUE  A  7th  Army 
ODCSOPS 

USAAREUE  Director  of  CED 
APO  New  York  09*103 

DR.  RALPH  DUS EK 
U.S.  ARMY  RESEARCH  INSTITUTE 
5001  EISENHOWER  AVENUE 
ALEXANDRIA,  VA  22333 

Dr.  Myron  Fischl 

U.S.  Army  Research  Institute  for  the 
Social  and  Behavioral  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Dr.  Harold  F.  O'Neil,  Jr. 

Attn:  PERI-OK 
Army  Research  Institute 
5001  Elsenhower  Avenue 
Alexandria,  VA  22333 

Mr.  Robert  Ross 

U.S.  Army  Research  Institute  for  the 
Social  and  Behavioral  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22 333 

Dr.  Robert  Saamor 

U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Commandant 

US  Army  Institute  of  Administration 
Attn:  Dr.  Sherrill 
FT  Benjamin  Harrison,  IN  46256 

Dr.  Frederick  Steinheiser 
U.  S.  Army  Reserch  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22 333 


Dr.  Joseph  Ward 
U.S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 
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Air  Force  Marines 


1  Air  Force  Human  Resources  Lab 
AFHRL/MPO 

Brooks  AFB,  TX  78235 

1  Air  University  Library 
AUL/LSE  76/4H3 
Maxwell  AFB,  AL  36112 

1  Dr.  Earl  A.  Alluisi 
HO.  AFHRL  (AFSC) 

Brooks  AFB.  TX  78235 

1  Dr.  Genevieve  Haddad 
Program  Manager 
Life  Sciences  Directorate 
AFOSR 

Bolling  AFB.  DC  20332 

1  Dr.  Ross  L.  Morgan  ( AFHRL/LR ) 
(Aright  -Patterson  AFB 
Ohio  45433 

1  Research  and  Measurment  Division 
Research  Branch,  AFMPC/MPCYPR 
Randolph  AFB.  TX  78148 

1  Dr.  Malcolm  Ree 
AFHRL/MP 

Brocks  AFB,  TX  78235 

1  Dr.  Marty  Rockway 
Technical  Director 
AFHRL (OT) 

Williams  AFB,  AZ  58224 

1  Jack  A.  Thorp,  Maj.,  USAF 
Life  Sciences  Directorate 
AFOSR 

Bolling  AFB,  DC  20332 


1  H.  William  Greenup 

Education  Advisor  (E03D 
Education  Center,  MCDEC 
Quantico,  VA  22134 

1  Director,  Office  of  Manpower  Utilization 
HQ,  Marine  Corps  (MPU) 

BCB,  Bldg.  2009 
Quantico,  VA  22134 

1  DR.  A.L.  SLAFKOSKY 

SCIENTIFIC  ADVISOR  (CODE  RD-1) 

HQ.  U.S.  MARINE  CORPS 
WASHINGTON,  DC  20380 
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Coast Guard 


Other  DcD 


1  Hr.  Thomas  A.  Warm 

U.  S.  Coast  Guard  Institute 
P.  0.  Substation  18 
Oklahoma  City,  OK  73169 

1  Dr.  Dexter  Fletoher 

ADVANCED  RESEARCH  PROJECTS  AGENCY 
1400  WILSON  BLVD. 

ARLINGTON,  VA  22209 

1  Dr.  William  Graham 
Testing  Directorate 
MEPCOM/MEPCT-P 
Ft.  Sheridan,  IL  60037 

1  Military  Assistant  for  Training  and 
Personnel  Technology 

Office  of  the  Under  Secretary  of  Defense 
for  Research  A  Engineering 
Room  3D 129,  The  Pentagon 
Washington,  DC  20301 

1  MAJOR  Wayne  Sellman,  USAF 

Office  of  the  Assistant  Secretary 
of  Defense  (MRA&L) 

3B930  The  Pentagon 
Washington,  DC  20301 

1  HEAD,  SECTION  ON  MEDICAL  EDUCATION 
UNIFORMED  SERVICES  UNIV.  OF  THE 
HEALTH  SCIENCES 
6917  ARLINGTON  ROAD 
BETHESDA,  MD  20014 


12  Defense  Technical  Information  Center 
Cameron  Station,  Bldg  5 
Alexandria,  VA  2231* 

Attn:  TC 
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Civil  Govt 


1  Dr.  Susan  Chipman 

Learning  and  Development 
National  Institute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20208 

1  Dr.  Joseph  I.  Llpaon 
SEDR  W-638 

National  Science  Foundation 
Washington,  DC  20550 

1  Dr.  John  Hays 

National  Institute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20208 

1  Dr.  Arthur  Helmed 

National  Intitute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20208 

1  Dr.  Andrew  R.  Molnar 
Science  Education  Dev. 
and  Research 

National  Science  Foundation 
Washington ,  DC  20550 

1  Personnel  RAD  Center 

Office  of  Personnel  Managnent 
1900  E  Street  NW 
Washington,  DC  20115 

1  Dr.  Vern  W.  UTry 
Personnel  RAD  Center 
Office  of  Personnel  Management 
1900  E  Street  NW 
Washington,  DC  20115 

1  Dr.  Frank  Withrow 

U.  S.  Office  of  Education 
100  Maryland  Ave.  SW 
Washington.  DC  20202 

1  Dr.  Joseph  L.  Young,  Director 
Memory  &  Cognitive  Processes 
National  Science  Foundation 
Washington ,  DC  20550 


Non  Govt 


1  Dr.  Erling  B.  Andersen 
Department  of  Statistics 
Studiestraede  6 
1155  Copenhagen 
DENMARK 

1  1  psychological  research  unit 

Dept,  of  Defense  (Army  Office) 
Campbell  Park  Offices 
Canberra  ACT  2600,  Australia 

1  Dr.  Isaac  Be jar 

Educational  Testing  Service 
Princeton,  NJ  08150 

1  Dr.  Werner  Birke 

DezWPs  im  Streitkraefteamt 
Fostfach  20  50  03 
D-5300  Bonn  2 
WEST  GERMANY 

1  Dr.  Nicholas  A.  Bond 
Dept,  of  Psychology 
Sacramento  State  College 
600  Jay  Street 
Sacramento,  CA  95819 

1  Dr.  David  G.  Bowers 

Institute  for  Social  Research 
University  of  Michigan 
P.  0.  Box  1218 
Ann  Arbor,  MI  18106 

1  Dr.  Robert  Brennan 

American  College  Testing  Programs 

P.  0.  Box  168 

Iowa  City,  IA  52210 

1  DR.  C.  VICTOR  BUNDERSON 
WICAT  INC 

UNIVERSITY  PLAZA,  SUITE  10 
1160  SO.  STATE  ST. 

OREM,  UT  81057 

1  Dr.  John  B.  Carroll 
Psychometric  Lab 
Unlv.  of  No.  Carolina 
Davie  Hall  0138 
Chapel  Hill,  NC  27511 
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Non  Govt 


1  Charles  Myers  Library 
Livingstone  House 
Livingstone  Road 
Stratford 
London  E15  2LJ 
ENGLAND 

1  Dr.  Kenneth  E.  Clark 

College  of  Arts  k  Sciences 
University  of  Rochester 
River  Campus  Station 
Rochester.  NT  14627 

1  Dr.  Norman  Cliff 
Dept,  of  Psychology 
Uhiv.  of  So.  California 
University  Park 
Los  Angeles,  CA  90007 

1  Dr.  William  E.  Cofftaan 

Director,  Iowa  Testing  Programs 
334  Lindquist  Center 
Uhiverslty  of  Iowa 
Iowa  City,  IA  52242 

1  Dr.  Meredith  P.  Crawford 

American  Psychological  Association 
1200  17th  Street,  N.W. 

Washington,  DC  20036 

1  Dr.  Hans  CTombag 

Education  Research  Center 
University  of  Leyden 
Bcerhaavelaan  2 
2334  EN  Leyden 
The  NETHERLANDS 

1  Dr.  Bnmanuel  Donchin 

Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 

1  LCOL  J.  C.  Eggenberger 

DIRECTORATE  OF  PERSONNEL  APPLIED  RESEARC 
NATIONAL  DEFENCE  HQ 
101  COLONEL  BY  DRIVE 
OTTAWA,  CANADA  K1A  0K2 


Non  Govt 


1  Dr.  Leonard  Feldt 

Lindquist  Center  for  Measurment 
University  of  Iowa 
Iowa  City,  IA  52242 

1  Dr.  Richard  L.  Ferguson 

The  American  College  Testing  Program 

P.O.  Box  168 

Iowa  City,  IA  52240 

1  Dr.  Victor  Fields 
Dept,  of  Psychology 
Montgomery  College 
Rockville,  MD  20850 

I  Univ.  Prof.  Dr.  Gerhard  Fischer 
Liebiggasse  5/3 
A  1010  Vienna 
AUSTRIA 

1  Professor  Donald  Fitzgerald 
University  of  New  England 
Armidale,  New  South  Hales  2351 
AUSTRALIA 

1  Dr.  Edwin  A.  Fleishman 

Advanced  Research  Resources  Organ. 
Suite  900 

4330  East  West  Highway 
Washington,  DC  20014 

1  Dr.  John  R.  Frederiksen 
Bolt  Beranek  &  Newman 
50  Moulton  Street 
Cambridge,  MA  02138 

1  DR.  ROBERT  GLASER 
LRDC 

UNIVERSITY  OF  PITTSBURGH 
3939  O'HARA  STREET 
PITTSBURGH,  PA  15213 

1  Dr.  Ron  Hambleton 
School  of  Education 
University  of  Masaechusetts 
Amherst,  MA  01002 
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Non  Govt 

Non  Govt 

J 

1 

Dr.  Chester  Harris 

1 

Dr.  Michael  Levine 

School  of  Education 

Department  of  Educational  Psychology 

Uhiversity  of  California 

210  Education  Bldg. 

Santa  Barbara.  CA  93106 

University  of  Illinois 

Champaign,  IL  61801 

1 

Dr.  Lloyd  Humphreys 

Department  of  Psychology 

University  of  Illinois 

Champaign,  IL  61820 

1 

1 

Dr.  Charles  Lewis 

Faeulteit  Socials  We tense happen 

Rijksuniversiteit  Groningen 

Oude  Boterlngestraat 

1 

Library 

Groningen 

} 

Hum RRO/ We stern  Division 

2785?  Berwick  Drive 

NETHERLANDS 

| 

j 

Carmel,  CA  93921 

1 

Dr.  Robert  Linn 

College  of  Education 

!  i 

Dr.  Steven  Hunka 

University  of  Illinois 

Department  of  Education 

University  of  Alberta 

Urbana,  IL  61801 

j 

Edmonton,  Alberta 

1 

Dr.  Frederick  M.  Lord 

CANADA 

Educational  Testing  Service 

Princeton,  NJ  08540 

I  i 

Dr.  Earl  Hunt 

Dept,  of  Psychology 

1 

Dr.  Gary  Marco 

Uhiversity  of  Washington 

Educational  Testing  Service 

Seattle.  WA  98105 

Princeton,  NJ  08450  j 

i 

Dr.  Huynh  Huynh 

1 

Dr.  Scott  Maxwell 

College  of  Education 

Department  of  Psychology 

University  of  South  Carolina 

Uhiversity  of  Houston 

Cbluabia,  SC  29208 

Houston,  TX  77004 

i 

Dr.  Douglas  H.  Jones 

1 

Dr.  Samuel  T.  Mayo 

An  T-255 

Loyola  Uhiversity  of  Chicago  j 

Educational  Testing  Service 

820  North  Michigan  Avenue 

Princeton,  NJ  08450 

Chicago,  IL  60611 

i 

Professor  John  A.  Keats 

1 

Dr.  James  A.  Paulson 

University  of  Newcastle 

Portland  State  Uhiversity 

AUSTRALIA  2308 

P.0.  Box  751 

Portland,  OR  97207 

i 

Dr.  Mazie  Knerr 

Litton-Mellonlcs 

1 

MR.  LUIGI  PETRULLO 

Box  1286 

2431  N.  EDGEVOOD  STREET  1 

; 

Springfield,  VA  22151 

ARLINGTON,  VA  22207 

i 

Mr.  Marlin  Kroger 

1 

DR.  DIANE  M.  RAMSEY-KLEE  | 

1117  Via  Goleta 

R-K  RESEARCH  A  SYSTEM  DESIGN  '  1 

u_ 

Palos  Verdes  Estates,  CA  90274 

3947  RIKEMONT  DRIVE  1 

MALIBU,  CA  90265  1 
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1  N INRAT  H.  L.  RAUCH 

P  II  A 

BUNDESH IN ISTER IUM  DER  VERTEIDIGUNG 

POSTFACH  1328 

0-63  BONN  1,  GERMANY 

1  Dr.  Mark  D.  Reckaae 

Educational  Psychology  Dept. 
University  of  Misaouri-Coluabis 
A  Hill  Hall 
Ccluabla,  MO  65211 

1  Dr.  Andrew  M.  Rose 

American  Institutes  for  Research 
1055  Thomas  Jefferson  St.  NW 
Washington ,  DC  20007 

1  Dr.  Leonard  L.  Rosen  bam ,  Chairman 
Department  of  Psychology 
Montgomery  College 
Rockville.  MD  20850 

1  Dr.  ETnst  Z.  Rothko pf 
Bell  Laboratories 
600  Mountain  Avenue 
Murray  Hill,  NJ  0797* 

1  Dr.  Lawrence  Rudner 
*03  Elm  Avenue 
Takoma  Park,  HD  20012 

1  Dr.  J.  Ryan 

Department  of  Education 
Uhiversity  of  South  Carolina 
Columbia,  SC  29208 

1  PROF.  FUMIKD  SAMEJIMA 
DEPT.  OF  PSYCHOLOGY 
UNIVERSITY  OF  TENNESSEE 
KNOXVILLE,  TN  37916 

1  Committee  on  Cognitive  Research 
%  Dr.  Lonnie  R.  Sherrod 
Social  Science  Research  Council 
605  Third  Avenue 
New  York,  NY  10016 


Non  Govt 


1  Dr.  Kaxuo  Shigemasu 
University  of  Ibhoku 
Department  of  Educational  Psychology 
Kawauchl,  Sendai  980 
JAPAN 

1  Dr.  Richard  Snow 
School  of  Education 
Stanford  University 
Stanford ,  CA  9*305 

1  Dr.  Robert  Sternberg 
Dept,  of  Psychology 
Yale  University 
Box  11A,  Yale  Station 
New  Haven,  CT  06520 

1  DR.  PATRICK  SUPPES 

INSTITUTE  FOR  MATHEMATICAL  STUDIES  IN 
THE  SOCIAL  SCIENCES 
STANFORD  UNIVERSITY 
STANFORD,  CA  9*305 

1  Dr.  Hariharan  Swan ina than 

Laboratory  of  Psychometric  and 
Evaluation  Research 
School  of  Education 
University  of  Massachusetts 
Amherst,  MA  01003 

1  Dr.  Brad  Synpaon 

Psychometric  Research  Group 
Educational  Testing  Service 
Princeton,  NJ  085*1 

1  Dr.  Kikuni  Tatauoka 

Computer  Based  Education  Research 
Laboratory 

252  Engineering  Research  Laboratory 
University  of  Illinois 
UTbana,  IL  61801 

1  Dr.  David  Ihlssen 

Department  of  Psychology 
Uhiversity  of  Kansas 
Lawrenoe,  KS  660** 
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1  Dr.  J.  Uilmtr 

Parceptrcnlcs,  Inc. 

6271  Variel  Avenue 
Mood  land  mile,  a  91 964 

1  Or.  Howard  Mainer 

Bureau  of  Social  science  Research 
1990  H  Street.  N.  W. 

Washington ,  DC  20036 

1  DR.  THOMAS  WALLSTfM 

PSYCHOMETRIC  LABORATORY 
DAVIE  HALL  01 3A 
UNIVERSITY  OF  NORTH  CAROL 
CHAPEL  HILL,  NC  27514 

1  Or.  David  J.  Weiss 
N660  Elliott  Hall 
University  of  Minnesota 
75  E.  River  Road 
Minneapolis,  MN  55455 

1  DR.  SUSAN  E.  WHITELY 
PSYCHOLOGY  DEPARTMENT 
UNIVERSITY  OF  KANSAS 
LAWRENCE.  KANSAS  66044 

1  Wolfgang  WlldgrUbe 
Streitkraeftesat 
Box  20  50  03 
D-5300  Bonn  2 
WEST  GERMANY 

1  Dr.  j.  Arthur  Ubodward 
De parts ent  of  Psychology 
University  of  California 
Los  Angeles,  CA  90024 

1  Dr.  Karl  linn 

Canter  for  reaearch  on  Learning 
and  Teaching 
Uhiverslty  of  Michigan 
Aon  Arbor,  MI  4810A 


l 


